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Why  GAO  Did  This  Study 

The  National  Defense  Authorization 
Act  for  Fiscal  Year  2012  includes  a 
provision  for  GAO  to  select,  assess, 
and  report  on  DOD  MAIS  programs 
annually  through  March  2018.  MAIS 
programs  are  intended  to  help  the 
department  sustain  its  key  operations. 
This  report:  (1)  evaluates  DOD’s 
implementation  of  statutory  reporting 
requirements  for  MAIS  programs 
experiencing  a  critical  change;  (2) 
describes  the  extent  to  which  selected 
MAIS  programs  have  changed  their 
planned  cost  and  schedule  estimates, 
and  met  performance  targets;  (3) 
assesses  the  extent  to  which  selected 
MAIS  programs  have  used  key  IT 
acquisition  best  practices,  including 
requirements  and  risk  management; 
and  (4)  determines  the  extent  to  which 
MAIS  programs  are  represented  on  the 
Dashboard.  GAO  compared 
information  on  programs  with  a  critical 
change  to  the  reporting  requirements. 
GAO  selected  three  programs  based 
on  factors,  such  as  representation  from 
each  military  service  (Air  Force,  Army, 
and  Navy),  identified  changes  to  cost, 
schedule,  and  performance,  and 
assessed  them  against  selected  best 
practices.  GAO  traced  the  programs  to 
the  Dashboard  and  reviewed  relevant 
processes. 

What  GAO  Recommends 

GAO  recommends,  among  others,  that 
DOD  examine  the  critical  change 
reporting  process  and  implement 
corrections  for  the  reports’  timeliness, 
and  address  weaknesses  with 
requirements  management,  and  add 
AT&L  as  a  responsible  organization  for 
MAIS  programs  to  the  Dashboard. 

DOD  concurred  with  all 
recommendations.  OMB  did  not  concur 
but  GAO  continues  to  believe  that 
improved  transparency  is  needed. 
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chac@gao.gov. 
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What  GAO  Found 

All  18  major  automated  information  system  (MAIS)  programs  that  experienced  a 
critical  change  to  program  cost,  schedule,  or  system  performance  targets 
submitted  complete  reports  to  Congress  that  contained  all  four  statutory 
elements,  but  16  programs  did  not  meet  the  requirement  to  report  to  Congress 
within  60  days  of  the  program  manager’s  submission  to  the  senior  Department  of 
Defense  (DOD)  official  that  led  to  the  critical  change  determination.  Of  the  16 
critical  change  reports  that  exceeded  the  60-days  to  report,  10  of  the  programs 
took  over  100  days.  Officials  said  that  60  days  is  too  short  to  perform  a  program 
evaluation.  Since  the  reports  were  not  always  timely,  Congress  may  not  have  the 
necessary  information  when  it  is  needed  to  make  decisions.  Finally,  the  DOD  did 
not  demonstrate  that  it  had  an  internal  control  to  ensure  that  MAIS  programs  not 
in  compliance  with  reporting  requirements  were  restricted  from  obligating  funds 
on  major  contracts  as  required  by  law. 


All  three  MAIS  programs  GAO  selected  to  review  experienced  changes  in  their 
cost  and  schedule  estimates,  and  one  program  did  not  fully  meet  its  technical 
performance  targets  (see  table). 


Table:  Extent  of  Changes  in  Cost  and  Schedule  Estimates,  and  the  Results  of  Technical 
Performance  Targets  (from  the  First  Acquisition  Program  Baseline  Estimate) 

Program 

Cost  increase 

Met  technical 

Schedule  change  performance  targets? 

Army 

program 

$368  million  (19%) 

3  months  (full 
deployment) 

Navy 

program 

$1.66  billion  (477%) 

13  years  9  months  (full 
deployment)8 

Air  Force 
program 

$129  million  (9%) 

6  months  (milestone  C) 

Source:  GAO  analysis  of  data  provided  by  DOD  officials.  |  GAO-16-336 

aDelay  was  attributed  to  a  major  change  in  project  scope  and  restructuring  of  the  program. 


The  three  selected  programs  implemented  all  seven  IT  acquisition  best  practices 
for  risk  management,  and  most  of  the  best  practices  were  implemented  for 
requirements  management:  the  Army  and  Navy  implemented  three  of  five  best 
practices  and  the  Air  Force  implemented  four  of  five  best  practices.  For  example, 
the  Army  program  did  not  adequately  manage  requirements  changes  and  ensure 
that  deliverables  were  in  alignment  with  requirements.  Until  the  programs  fully 
implement  best  practices  for  requirements  management,  management  of 
development  efforts  will  likely  be  impaired. 

As  of  October  2015,  all  appropriate  programs  were  represented  on  the  Federal  IT 
Dashboard  (Dashboard)  as  required  by  the  Office  of  Management  and  Budget 
(OMB);  however,  the  organization  responsible  for  performance  of  MAIS 
programs  was  not  provided.  Specifically,  DOD’s  Chief  Information  Officer  is 
shown  as  the  responsible  party  because  OMB  requires  this,  but  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (AT&L)  has 
overall  responsibility  for  the  MAIS  programs.  Therefore,  users  of  the  Dashboard 
are  unaware  that  AT&L  is  the  responsible  organization  and,  thus,  public 
accountability  of  the  MAIS  programs  is  decreased. 
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U.S.  GOVERNMENT  ACCOUNTABILITY  OFFICE 


March  30,  2016 
Congressional  Committees 

The  Department  of  Defense  (DOD)  is  one  of  the  largest  and  most 
complex  organizations  in  the  world.  To  meet  its  mission  to  protect  the 
security  of  our  nation  and  to  deter  war,  it  relies  heavily  on  the  use  of 
information  technology  (IT)  to  support  our  warfighters.  In  this  regard, 
according  to  DOD’s  IT  investment  portfolio  for  fiscal  year  2015,  the 
department  spent  approximately  $30  billion  for  IT  investments.1  Of  this 
amount,  approximately  $3.7  billion  was  spent  on  major  automated 
information  system  (MAIS)  programs,  which  are  intended  to  help  the 
department  sustain  its  key  operations.  These  include  communications, 
business,  and  command  and  control  systems  that  provide  department 
and  component  officials  with  access  to  information  to  organize,  plan, 
direct,  and  monitor  mission  operations. 

DOD  IT  investments  that  fall  within  one  of  the  following  categories  are 
designated  as  a  MAIS  program  when:  (1)  program  costs  in  any  single 
year  exceed  $40  million,  (2)  total  program  acquisition  costs  exceed  $165 
million,  or  (3)  total  life-cycle  costs  exceed  $520  million.2  The  Secretary  of 
Defense  can  also  use  discretion  to  designate  a  program  as  a  MAIS  if  it 
does  not  meet  these  cost  thresholds.  In  addition,  MAIS  programs  must 
comply  with  certain  annual  and  quarterly  reporting  requirements  identified 
in  statute.3 

The  National  Defense  Authorization  Act  for  Fiscal  Year  2012  includes  a 
provision  that  we  select,  assess,  and  report  on  DOD  MAIS  programs 
annually  through  March  201 8. 4  In  addition,  Senate  Report  113-176 
accompanying  S.  2410  includes  a  provision  that  we  evaluate  DOD’s 


1  DOD’s  IT  investment  portfolio  identifies  all  of  its  IT  investments  and  associated  costs 
within  the  department  and  its  components. 

2DOD  Instruction  5000.02,  Operation  of  the  Defense  Acquisition  System,  January  7,  201 5. 

310  U.S.C.  §§  2445b  and  2445c. 

4Pub.  L.  No.  112-81  §  1078  (Dec.  31, 2011).  Under  this  provision  we  report  on  these 
assessments  no  later  than  March  30  of  each  year  from  2013  through  2018. 
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implementation  of  statutory  reporting  requirements  for  MAIS  programs 
experiencing  a  critical  change.  A  critical  change  must  be  declared  if  the 
program  has  experienced,  among  other  things,  a  schedule  delay  of  1  year 
or  more,  a  full  life-cycle  cost  increase  of  25  percent  or  more  over  the 
original  estimate,  or  a  change  that  will  undermine  the  system’s  ability  to 
perform  as  intended.  The  Carl  Levin  and  Howard  P.  “Buck”  McKeon 
National  Defense  Authorization  Act  for  Fiscal  Year  2015  mandated  that 
agencies,  including  DOD,  enhance  transparency  and  risk  management  of 
their  major  IT  investments  through  the  reporting  of  performance 
information  on  the  Federal  IT  Dashboard,  a  website  that  allows 
stakeholders  and  the  public  to  view  details  on  the  effectiveness  of 
government  IT  programs.5 

Our  objectives  for  this  review  were  to:  (1)  evaluate  DOD’s  implementation 
of  statutory  reporting  requirements  for  MAIS  programs  experiencing  a 
critical  change;  (2)  describe  the  extent  to  which  selected  MAIS  programs 
have  changed  their  planned  cost  and  schedule  estimates,  and  met 
performance  targets;  (3)  assess  the  extent  to  which  selected  MAIS 
programs  have  used  key  IT  acquisition  best  practices,  including 
requirements  and  risk  management;  and  (4)  determine  the  extent  to 
which  MAIS  programs  are  represented  on  the  Federal  IT  Dashboard. 

To  accomplish  the  first  objective,  we  identified  18  of  39  MAIS  programs 
that  experienced  a  critical  change  and  compared  their  status  reports 
submitted  to  Congress  to  statutory  reporting  requirements  to  determine 
whether  gaps  exist.  We  also  interviewed  DOD  officials  responsible  for  the 
quality  of  the  data  and  assessed  their  procedures  for  maintaining  its 
accuracy  and  completeness  for  MAIS  programs  that  experienced  a  critical 
change  between  December  2008  and  June  2014,  as  well  as  factors 
impacting  the  timelines  in  delivering  these  critical  change  reports  to 
Congress  and  the  processes  used  to  ensure  that  appropriated  funds  were 
not  being  obligated  on  major  contracts  if  prohibited  under  law. 

To  address  the  second  and  third  objectives,  we  selected  3  programs  by 
identifying  them  from  the  MAIS  population  of  39  programs  that  met 
several  criteria,  such  as  programs  with  a  baseline  that  could  be  used  as  a 
reference  point  for  evaluating  cost  and  schedule  characteristics.  We 
selected  the  Army’s  Tactical  Mission  Command  (TMC),  the  Navy’s 


540  U.S.C.  §  11302(c)(3)(B). 
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Common  Aviation  Command  and  Control  System  (CAC2S),  and  the  Air 
Force’s  Defense  Enterprise  Accounting  and  Management  System 
(DEAMS). 

To  determine  the  extent  that  selected  program  estimates  changed,  we 
compared  their  best  or  objective  cost  and  schedule  estimates  established 
in  the  first  acguisition  baseline  estimate  (where  available)  to  the  latest 
total  life-cycle  estimates.6  To  determine  whether  technical  performance 
targets  were  met,  we  compared  each  program’s  system  performance 
targets7  against  actual  performance  data,  and  reviewed  the  results  of 
assessments  conducted  on  the  systems.  We  then  aggregated  and 
summarized  the  results  of  our  analyses  across  the  programs. 

To  determine  the  extent  that  best  practices  were  used,  we  identified  key 
risk  management  and  reguirements  management  practices  from  the 
Software  Engineering  Institute’s  Capability  Maturity  Model®  Integration 
for  Acguisition  and  assessed  each  program  against  these  and  other 
criteria.8  We  analyzed  documents,  such  as  risk  register  logs  and  risk 
management  plans,  and  compared  them  to  each  program’s  processes 
and  practices.  We  interviewed  program  officials  to  obtain  additional 
information  on  the  processes  and  practices  used. 

To  address  the  fourth  objective,  we  compared  all  39  MAIS  programs  to 
the  IT  Dashboard  website  to  identify  reporting  gaps.  We  also  evaluated 
the  process  the  department  used  to  determine  and  submit  status  and 
performance  information  to  the  website.  We  interviewed  officials  from  the 
DOD  Office  of  the  Chief  Information  Officer  (CIO),  Office  of  the  Under 
Secretary  of  Defense  for  Acguisition,  Technology,  and  Logistics  (AT&L), 


6An  estimate  in  then-year  dollars  includes  the  effects  of  economic  inflation.  The  first 
acquisition  program  baseline  is  established  after  the  program  has  assessed  the  viability  of 
various  technologies  and  refined  user  requirements  to  identify  the  most  appropriate 
technology  solution  that  demonstrates  that  it  can  meet  users’  needs.  The  Defense 
Acquisition  Guidebook  (which  complements  and  further  explains  DOD’s  acquisition 
policies  and  process)  refers  to  a  program’s  best  cost  and  schedule  estimates  as  objective 
estimates. 

7Program  system  targets  are  performance  attributes  of  a  system  considered  critical  or 
essential  to  the  development  of  an  effective  military  system.  These  are  also  known  as  key 
performance  parameters. 

8Software  Engineering  Institute,  Capability  Maturity  Model®  Integration  for  Acquisition, 
Version  1.3  (Pittsburgh,  Pa.:  November  2010). 
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and  the  Office  of  Management  and  Budget  (OMB)  to  obtain  their 
perspectives  on  the  quality  of  reporting  of  DOD  MAIS  programs. 

We  conducted  this  performance  audit  from  April  201 5  to  March  201 6  in 
accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives.  See  appendix  I  for  a  more 
detailed  discussion  of  our  objectives,  scope,  and  methodology. 


Background 


DOD’s  organizational  structure  includes  the  Office  of  the  Secretary  of 
Defense,  the  Joint  Chiefs  of  Staff,  the  military  departments,  numerous 
defense  agencies  and  field  activities,  and  various  unified  combatant 
commands  that  contribute  to  the  oversight  of  DOD’s  acquisition 
programs.  Figure  1  provides  a  simplified  depiction  of  DOD’s 
organizational  structure. 


Figure  1:  Simplified  DOD  Organizational  Structure 


Source:  GAO  analysis  based  on  Department  of  Defense  data.  |  GAO-16-336 


The  Under  Secretary  of  Defense  for  AT&L  serves  as  the  Defense 
Acquisition  Executive  and  the  Under  Secretary  has  responsibility  for 


Page  4 


GAO-16-336  Major  Automated  Information  Systems 


oversight  of  MAIS  acquisition  programs.  AT&L  has  policy  and  procedural 
authority  for  the  defense  acquisition  system,  which  establishes  the  steps 
that  DOD  programs  generally  take  as  DOD  plans,  acquires,  deploys, 
operates,  and  maintains  its  IT  systems  (discussed  in  more  detail  following 
this  section).  Additionally,  AT&L  is  the  principal  acquisition  official  of  the 
department  and  is  the  acquisition  advisor  to  the  Secretary  of  Defense. 
AT&L’s  authority  includes  directing  the  military  services  and  defense 
agencies  on  acquisition  matters  and  making  milestone  decisions  for  MAIS 
programs.  AT&L  can  delegate  decision  authority  for  MAIS  programs  to  a 
component  head  who  may  further  delegate  the  authority  to  the 
component  acquisition  executive.9 

DOD’s  CIO  is  the  Principal  Staff  Assistant  and  senior  IT  advisor  to  the 
Secretary  of  Defense.  This  role  includes  overseeing  many  national 
security  and  defense  business  systems  and  managing  information 
resources.  The  CIO  coordinates  with  AT&L  to  develop  and  maintain  a 
process  for  assessing  and  managing  the  risks  related  to  the  department’s 
IT  acquisitions,  including  MAIS  programs. 


DOD’s  Acquisition 
Guidance  and  Framework 
for  Managing  MAIS 
Programs 


The  Department  of  Defense  Instruction  5000. 0210  establishes  policy  for 
the  management  of  all  DOD  acquisition  programs.  In  January  2015,  DOD 
updated  these  guidelines  which  outline  the  framework  for  MAIS 
programs.  This  framework  consists  of  six  models  for  acquiring  and 
deploying  a  program,  including  two  hybrid  models  that  each  describe  how 
a  program  may  be  structured  based  on  the  type  of  product  being  acquired 
(e.g.,  software-intensive  programs  and  hardware-intensive  programs).  A 
generic  acquisition  model  that  shows  all  of  the  program  life-cycle  phases 
and  key  decision  points  is  shown  in  figure  2  and  described  following  the 
figure. 


9DOD  Instruction  5000.02,  Enclosure  1. 

10DOD  Instruction  5000.02,  Operation  of  the  Defense  Acquisition  System,  January  7, 
2015. 
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Figure  2:  Generic  Acquisition  Model  from  the  Defense  Acquisition  System 
Framework 
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Source:  GAO  analysis  based  on  Department  of  Defense  data.  |  GAO-16-336 


•  Materiel  solution  analysis:  Refine  the  initial  system  solution 
(concept)  and  create  a  strategy  for  acquiring  the  solution.  A 
decision — referred  to  as  milestone  A — is  made  at  the  end  of  this 
phase  to  authorize  entry  into  the  technology  maturation  and  risk 
reduction  phase. 

•  Technology  maturation  and  risk:  Determine  the  preferred 
technology  solution  and  validate  that  it  is  affordable,  satisfies  program 
requirements,  and  has  acceptable  technical  risk.  A  decision — referred 
to  as  milestone  B — is  made  at  the  end  of  this  phase  to  authorize  entry 
of  the  program  into  the  engineering  and  manufacturing  development 
phase  and  award  development  contracts.  An  acquisition  program 
baseline11  is  first  established  at  the  milestone  B  decision  point  or  at 
program  initiation,  whichever  occurs  later.  A  program’s  first  acquisition 


11  The  acquisition  program  baseline  is  developed  by  the  program  manager  before  the 
initiation  of  a  program  for  all  acquisition  programs  and  depicts  the  current  condition  of  a 
program.  The  plan  (which  must  be  approved  by  the  Milestone  Decision  Authority)  states 
the  threshold  and  objective  values  for  the  cost,  schedule  and  performance  requirements 
fora  program.  An  acquisition  program  baseline  is  required  for  each  increment  and  block 
for  a  MAIS  or  Major  Defense  Acquisition  Program.  Any  deviations  from  an  approved 
acquisition  program  baseline  have  to  be  documented. 
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program  baseline  contains  the  original  life-cycle  cost  estimate  (which 
includes  acquisition  and  operations  and  maintenance  costs),  the 
schedule  estimate  (which  consists  of  major  milestones  and  decision 
points),  and  performance  parameters  that  were  approved  for  that 
program  by  the  milestone  decision  authority.  The  first  acquisition 
program  baseline  is  established  after  the  program  has  refined  user 
requirements  and  identified  the  most  appropriate  technology  solution 
that  demonstrates  that  it  can  meet  users’  needs. 

•  Engineering  and  manufacturing  development:  Develop  a  system 
and  demonstrate  through  testing  that  the  system  meets  all  program 
requirements.  A  decision — referred  to  as  milestone  C — is  made  during 
this  phase  to  authorize  entry  of  the  system  into  the  production  and 
deployment  phase  or  into  limited  deployment  in  support  of  operational 
testing. 

•  Production  and  deployment:  Achieve  an  operational  capability  that 
meets  program  requirements,  as  verified  through  independent 
operational  tests  and  evaluation,  and  implement  the  system  at  all 
applicable  locations. 

•  Operations  and  support:  Operationally  sustain  the  system  in  the 
most  cost-effective  manner  over  its  life  cycle. 


Statutory  Reporting 
Requirements  for  MAIS 
Programs 


MAIS  programs  enable  DOD  to  organize,  plan,  direct  and  monitor 
important  mission  operations.  As  previously  mentioned,  MAIS  programs 
must  comply  with  certain  annual  and  quarterly  reporting  requirements 
identified  in  statute.12  Each  calendar  year,  DOD  must  submit  to  Congress 
budget  justification  documents  on  each  MAIS  program,  including 
information  on  cost,  schedule,  and  performance.  Specifically,  these 
programs  must  report,  among  other  things,  on  the  development  and 
implementation  schedules  and  total  acquisition  and  full  life-cycle  cost 
estimates  and  provide  a  summary  of  the  key  performance  parameters  for 
each  program.  DOD  must  also  provide  a  summary  of  any  significant 
changes  to  information  previously  provided  for  each  program. 

Moreover,  on  a  quarterly  basis,  the  program  manager  for  each  MAIS 
program  is  required  to  provide  the  senior  DOD  official  responsible  for  the 


1210  U.S.C.,  §§  2445b  and  2445c. 
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program  a  written  report  that  identifies  any  variance  in  the  program’s  cost, 
schedule,  or  performance.  Depending  on  the  determination  after 
reviewing  the  variances  identified  in  the  quarterly  report,  the  senior  DOD 
official  responsible  for  the  program  must  notify  the  congressional  defense 
committees  of  any  programs  that  have  experienced  either  a  significant  or 
critical  change.  During  our  review,  MAIS  programs  were  required  to 
comply  with  the  following  reporting  requirements: 

•  Significant  change.  A  significant  change  must  be  declared  if  a 
program  experienced  a  schedule  delay  of  more  than  6  months  but 
less  than  a  year;  estimated  total  acquisition  or  full  life-cycle  cost  for 
the  program  has  increased  by  at  least  15  percent  but  less  than  25 
percent;  or  there  has  been  a  significant  adverse  change  in  the 
expected  performance  of  the  system.  If  such  an  event  occurs,  the 
senior  DOD  official  responsible  for  the  program  must  notify  the 
congressional  defense  committees  in  writing  no  later  than  45  days 
after  receiving  the  quarterly  report  from  the  program  manager. 

•  Critical  change.  A  critical  change  must  be  declared  if  a  program 
failed  to  achieve  a  full  deployment  decision  within  5  years  after  the 
milestone  A  decision  or,  if  there  was  no  milestone  A  decision,  the  date 
when  the  preferred  alternative  was  selected  for  the  program; 
experienced  a  schedule  delay  of  1  year  or  more;  experienced  an 
estimated  total  acquisition  or  full  life-cycle  cost  increase  of  25  percent 
or  more  over  the  original  estimate;  or  experienced  a  change  in  the 
expected  performance  of  the  system  that  will  undermine  the  ability  of 
the  system  to  perform  as  intended.  If  such  an  event  occurs,  the  senior 
DOD  official  responsible  for  the  program  must  carry  out  an  evaluation 
and  submit  a  critical  change  report  to  the  congressional  defense 
committees  no  later  than  60  days  after  receiving  the  quarterly  report.13 


1  ^ 

In  certain  cases,  DOD  does  not  need  to  carry  out  an  evaluation  and  submit  a  report. 
Specifically,  if  the  senior  DOD  official  with  milestone  decision  authority  determines  that  a 
critical  change  is  primarily  due  to  an  extension  of  a  program  and  involves  minimal 
developmental  risk,  the  official  may  instead  submit  to  the  congressional  defense 
committees  a  certification  that  the  official  has  made  those  determinations.  This 
certification  must  be  submitted  within  45  days  after  receiving  the  quarterly  report.  DOD 
guidance  states  that,  in  cases  where  the  senior  official  is  not  within  the  Office  of  the 
Secretary  of  Defense,  he/she  should  provide  the  signed  report  and  draft  transmittal  letters 
addressed  to  the  congressional  defense  committees  to  the  cognizant  official  for  transmittal 
to  Congress,  no  later  than  5  working  days  before  expiration  of  the  60-day  period. 
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Since  the  December  19,  2014,  enactment  of  the  Carl  Levin  and  Howard 
P.  “Buck”  McKeon  National  Defense  Authorization  Act  for  Fiscal  Year 
2015,  MAIS  programs  are  now  required  to  declare  a  significant  change — 
instead  of  a  critical  change — if  they  fail  to  achieve  a  full  deployment 
decision  within  5  years  after  the  milestone  A  decision,  the  date  when  the 
preferred  alternative  was  selected  for  the  program  (excluding  any  time 
during  which  program  activity  is  delayed  as  a  result  of  a  bid  protest). 

More  recently,  the  National  Defense  Authorization  Act  for  Fiscal  Year 
201 614  directed  the  Secretary  of  Defense  to  issue  guidance  for  MAIS 
programs  to  establish  an  acquisition  baseline  within  2  years  after  program 
initiation.  This  statute  provides  a  response  to  a  recommendation  we  made 
in  our  last  annual  report15  on  MAIS  programs.  In  particular,  we  found  that 
these  programs  spent,  on  average,  more  than  5  years  and  $450  million 
prior  to  establishing  baselines.  We  noted  that  programs  that  have  not 
established  baselines  were  subject  to  less  oversight  and  could  not  be 
measured  against  cost,  schedule,  and  performance  targets.  Also,  the 
propensity  to  carry  out  MAIS  programs  for  multiple  years  prior  to 
committing  to  baselines  is  inconsistent  with  incremental  and  rapid 
development  as  called  for  in  federal  law  and  GAO’s  IT  management  best 
practices.16  Accordingly,  we  recommended  that  these  programs  be 
baselined  within  2  years;  for  which  DOD  partially  concurred.  We 
maintained  that  establishing  baselines  within  2  years  would  improve 
outcomes  and  increase  accountability. 


14National  Defense  Authorization  Act  for  Fiscal  Year  2016,  Pub.  L.  No.  1 14-92  §  883(e) 
(Nov.  25.  2015). 

15GAO-1 5-282. 

16See  41  U.S.C.  §  2308;  see  also  48  C.F.R.  §  39.103  (Federal  Acquisition  Regulation); 
OMB,  Management  of  Federal  Information  Resources,  Circular  No.  A-130  Revised.  Also, 
40  U.S.C.  §  1 131 9(b)(1  )(B)(ii)  requires  the  CIO  of  each  covered  agency  to  certify  that  IT 
investments  are  adequately  implementing  incremental  development  as  defined  in  OMB’s 
capital  planning  guidance. 
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OMB’s  IT  Dashboard 
Codified  to  Provide 
Transparency  into  the 
Performance  of  Agency  IT 
Investments 


DOD’s  CIO,  along  with  other  agencies,  must  report  on  the  progress  of  its 
IT  investments,  including  MAIS  programs,  on  a  public  website  known  as 
the  IT  Dashboard.17  OMB  established  this  website  in  June  2009  to 
improve  the  transparency  and  oversight  of  agencies’  investments.  The 
Dashboard  visually  displays  federal  agencies’  cost,  schedule,  and 
performance  data  for  over  700  major  federal  investments  at  26  federal 
agencies.18  It  also  includes  a  risk  rating  that  is  to  be  performed  by  agency 
CIOs.  According  to  OMB,  these  data  are  intended  to  provide  a  near-real- 
time  perspective  on  the  performance  of  these  investments.19 

The  public  display  of  agency  data  is  intended  to  allow  OMB;  other 
oversight  bodies,  including  Congress;  and  the  general  public  to  hold 
federal  agencies  accountable  for  their  progress  and  results.  In  August 
201 1 ,  OMB  issued  guidance  that  stated,  among  other  things,  that  agency 
CIO’s  shall  be  held  accountable  for  the  performance  of  IT  investments. 
The  Dashboard  presents  performance  ratings  for  individual  investments 
using  metrics  that  OMB  has  defined — cost,  schedule,  and  CIO  evaluation. 
If  OMB  or  the  agency  CIO  determine  the  reported  data  is  not  timely  or 
reliable,  the  CIO  must  notify  OMB  and  establish  within  30  days  of  this 
determination  an  improvement  program  and  the  progress  the  agency  is 
making.  According  to  OMB,  the  addition  of  CIO  names  and  photos  on  the 
website  is  intended  to  highlight  this  accountability  and  link  the 
Dashboard’s  reporting  on  investment  performance. 

In  order  to  enhance  transparency  and  improve  risk  management  of 
federal  IT  acquisitions,  Congress  codified  the  Dashboard  reporting 
process  through  key  provisions,  known  as  the  Federal  Information 
Technology  Acquisition  Reform  provisions  in  the  Carl  Levin  and  Howard 


17http://www.  itdashboard.gov. 

1 R 

The  26  federal  agencies  are  the  Departments  of  Agriculture,  Commerce,  Defense, 
Education,  Energy,  Health  and  Human  Services,  Homeland  Security,  Housing  and  Urban 
Development,  Justice,  Labor,  State,  the  Interior,  the  Treasury,  Transportation,  Veterans 
Affairs;  the  Environmental  Protection  Agency;  General  Services  Administration;  National 
Aeronautics  and  Space  Administration;  National  Archives  and  Records  Administration; 
National  Science  Foundation;  Nuclear  Regulatory  Commission;  Office  of  Personnel 
Management;  Small  Business  Administration;  Social  Security  Administration;  U.S.  Agency 
for  International  Development;  and  U.S.  Army  Corps  of  Engineers. 

1 Q 

In  March  2014,  DOD  issued  a  memorandum  that  revised  its  CIO  ratings  process  for  the 
IT  Dashboard.  The  memorandum  calls  for  semiannual  updates  to  be  submitted 
concurrently  with  the  IT  budget. 
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P.  “Buck”  McKeon  National  Defense  Authorization  Act  for  Fiscal  Year 
201 5.20 


Best  Practices  for 
Managing  IT  Acquisition 
Programs 


Entities  such  as  the  Project  Management  Institute,  the  Software 
Engineering  Institute  at  Carnegie  Mellon  University,  and  GAO  have 
developed  and  identified  best  practices  to  help  guide  organizations  to 
effectively  plan  and  manage  their  acquisitions  of  major  IT  systems,  such 
as  MAIS  programs.  Our  prior  reviews  have  shown  that  properly  applying 
such  practices  can  significantly  increase  the  likelihood  of  delivering 
promised  system  capabilities  on  time  and  within  budget.  These  practices 
include,  but  are  not  limited  to: 


•  Requirements  management:  Requirements  establish  what  the 
system  is  to  do,  how  well  it  is  to  do  it,  and  how  it  is  to  interact  with 
other  systems.  Appropriate  requirements  development  involves 
eliciting  and  developing  customer  and  stakeholder  requirements,  and 
analyzing  them  to  ensure  that  they  will  meet  users’  needs  and 
expectations.  It  also  consists  of  validating  requirements  as  the  system 
is  being  developed  to  ensure  that  the  final  systems  to  be  deployed  will 
perform  as  intended  in  an  operational  environment. 

•  Risk  management:  A  process  for  anticipating  problems  and  taking 
appropriate  steps  to  mitigate  risks  and  minimize  their  impact  on 
program  commitments.  It  involves  identifying  and  documenting  risks, 
categorizing  them  based  on  their  estimated  impact,  prioritizing  them, 
developing  risk  mitigation  strategies,  and  tracking  progress  in 
executing  the  strategies. 


on 

Federal  Information  Technology  Acquisition  Reform  provisions  of  the  Carl  Levin  and 
Howard  P.  “Buck”  McKeon  National  Defense  Authorization  Act  for  Fiscal  Year  2015,  Pub. 
L.  No.  113-291,  div.  A,  title  VIII,  subtitle  D,  §  832,  128  Stat.  3292,  3440-3442  (Dec.  19, 
2014). 
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All  Programs  with  a 
Critical  Change 
Reported  Required 
Elements,  but  Most 
Missed  Deadlines 
and  Obligations  Were 
Not  Monitored 


According  to  statute,  for  programs  that  declare  a  critical  change,  the 
report  that  is  submitted  to  Congress  must  include  a  written  certification 
stating  that:21 

•  the  automated  information  system  or  IT  investment  to  be  acquired  is 
essential  to  the  national  security  or  to  the  efficient  management  of  the 
DOD; 

•  there  is  no  alternative  to  the  system  or  IT  investment  which  will 
provide  equal  or  greater  capability  at  less  cost; 

•  the  new  estimates  of  the  costs,  schedule,  and  performance 
parameters  have  been  determined,  with  the  concurrence  of  the 
Director  of  Cost  Assessment  and  Program  Evaluation,  to  be 
reasonable;  and 

•  the  management  structure  for  the  program  is  adequate  to  manage 
and  control  program  costs. 


All  18  MAIS  critical  change  reports  in  our  review  contained  the  required 
elements. 


In  addition,  this  report  must  be  prepared  and  submitted  to  Congress  no 
later  than  60  days  after  the  senior  DOD  official  responsible  for  the 
program  receives  the  quarterly  report  from  the  program  manager  that 
leads  to  the  determination  that  a  critical  change  event  has  occurred. 
Programs  that  do  not  submit  the  report  to  Congress  within  the  60-day 
period  are  statutorily  prohibited  from  obligating  appropriated  funds  for  any 
major  contract  until  the  date  that  Congress  receives  the  report.  Further,  if 
a  MAIS  program  violates  the  statutory  prohibition  against  obligations,  it 
will  also  violate  the  Antideficiency  Act.22  This  act  prohibits  an  officer  or 
employee  of  the  United  States  government  from  making  or  authorizing  an 
expenditure  or  obligation  in  excess  of  or  in  advance  of  available 
appropriations.  The  Antideficiency  Act  also  requires  that  an  appropriation 
must  be  available  for  an  agency  to  incur  an  obligation.  Thus,  if  DOD 
incurs  an  obligation  against  an  appropriation  that  is  not  legally  available, 
the  department  has  violated  the  act.  Violating  the  Antideficiency  Act 
would  require  the  Secretary  of  Defense  to  immediately  report  to  the 
President  and  Congress  all  relevant  facts  and  a  statement  of  actions 
taken. 


21 10  U.S.C.  §  2445c  (f). 
2231  U.S.C.  §  1341. 
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Of  the  18  MAIS  programs  experiencing  a  critical  change,  most  exceeded 
the  60-day  reporting  requirement,  several  by  a  substantial  amount. 
Specifically,  16  exceeded  the  60-day  reporting  requirement  and  10  of 
those  programs  took  over  100  days  to  report.  Of  the  10  programs,  5  of 
the  programs  took  over  200  days  to  report.  Two  programs — Teleport  Gen 
I/ll  and  General  Fund  Enterprise  Business  Systems — delivered  their 
reports  to  Congress  within  the  60-day  requirement.  Figure  3  shows  the 
extent  that  programs  met  or  exceeded  the  60-day  reporting  requirement. 
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Figure  3:  Extent  that  MAIS  Programs  Met  or  Exceeded  the  60-day  Reporting  Requirement 
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Officials  from  several  programs  provided  various  reasons  for  why  they 
delayed  submitting  their  critical  change  reports.  For  example: 

e  Mission  Planning  Systems  #2  submitted  a  report  to  the  Office  of  the 
Secretary  of  Defense  on  time  but  it  took  29  days  to  transmit  the 
package  to  Congress  and  it  was  3  days  late. 
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•  CAC2S  program  was  late  because  of  the  need  to  conduct  an 
independent  assessment  that  was  directed  by  DOD. 

•  The  Expeditionary  Combat  Support  System  program  had  delays  due 
to  changes  that  were  made  to  the  size  and  complexity  of  its  originally 
scoped  effort  and  contracting  process  that,  in  turn,  required  additional 
updates.23  The  update  triggered  a  process  to  re-evaluate  the  revised 
strategy. 

According  to  a  DOD  AT&L  official,  60  days  is  too  short  to  perform  a 
program  evaluation  and  achieve  all  the  coordination  necessary  for  an 
important  communication  with  Congress.  The  official  also  said  the 
addition  of  the  Office  of  Cost  Assessment  and  Program  Evaluation 
requirement  to  review  and  approve  the  reports,  as  mandated  by  the 
Weapon  Systems  Acquisition  Reform  Act  of  2009, 24  consumes  much  of 
the  60-day  allotment  time.  A  Cost  Assessment  and  Program  Evaluation 
official  noted  that  reviews  often  exceed  the  60-day  period  because,  most 
notably,  the  significant  amount  of  time  needed  to  collect  and  develop 
comprehensive  information  used  to  determine  a  program’s  cost.  The 
official  added  that  DOD  is  working  to  strengthen  the  data  collection  efforts 
to  improve  the  ability  of  the  Office  of  Cost  Assessment  and  Program 
Evaluation  to  complete  its  evaluation  such  as  reviewing  the  basis  for  the 
revised  cost  and  schedule  estimates.  However,  the  official  noted  that 
there  has  been  no  overall  evaluation  or  study  on  the  cause  for  delays. 

While  it  may  be  possible  that  60  days  is  too  short  of  a  time  frame  for 
submitting  reports,  without  understanding  the  cause  for  the  delays  DOD  is 
not  in  a  position  to  state  what  time  frame  would  be  feasible.  Further,  the 
fact  that  two  programs  were  able  to  submit  reports  in  a  timely  manner 
suggests  that  60  days  is  achievable.  Until  DOD  ascertains  the  cause  for 
the  delays  and  implements  corrective  actions,  the  reports  may  continue  to 
be  delivered  in  a  manner  which  may  impact  the  timeliness  of  information 
considered  by  Congress  in  making  oversight  and  funding  decisions  for 
MAIS  programs.  This  may  also  affect  budget  and  other  strategic 
decisions  on  how  and  what  programs  to  prioritize. 

Further,  DOD  does  not  have  a  mechanism  to  monitor  and  ensure  that 
MAIS  programs  with  late  reports  were  restricted  as  required  by  law  from 


oo 

The  Expeditionary  Combat  Support  System  program  was  cancelled  in  December  2012. 
2410  U.S.C.  §  2334(a)(6)(A)(iv). 
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obligating  funds  on  major  contracts  prior  to  Congress  receiving  the  report. 
This  mechanism  is  especially  important  because  of  the  potential  for 
violating  the  Antideficiency  Act.  Although  DOD  states  in  its  guidance  that 
program  managers  should  not  obligate  any  funds  during  the  entire  period 
in  which  the  report  is  being  prepared,  DOD  does  not  currently  have  a  way 
to  monitor  this. 

According  to  a  DOD  AT&L  official,  DOD  is  not  required  by  statute, 
regulation,  or  guidance  to  collect  the  information  for  monitoring  purposes. 
However,  our  guidance  on  internal  controls  for  federal  agencies  states 
that  agency  management  should  establish  a  baseline  to  monitor  the 
current  state  of  a  control  system.  Once  established,  management  should 
monitor  the  agencies’  internal  control  system  through  ongoing  monitoring 
and  separate  evaluations  25  DOD  does  not  have  a  management  internal 
control  to  monitor  the  system  and  evaluate  whether  programs  are 
complying  with  the  DOD  guidance.  Instead,  DOD  relies  on  the  programs 
to  act  in  accordance  with  the  law.  This  official  said  the  need  to  obligate 
funds  on  major  contracts  should  be  a  driver  for  programs  to  submit  their 
reports  to  Congress  as  expeditiously  as  possible.  However,  with  so  many 
programs  submitting  the  critical  change  reports  well  after  the  60-day 
period,  there  is  a  risk  that  programs  could  potentially  violate  the 
prohibition  on  obligations,  and  thus,  in  addition,  the  Antideficiency  Act. 


Selected  MAIS 
Programs  Had 
Varying  Cost  and 
Schedule  Estimate 
Changes,  One  Did 
Not  Meet  Its  Technical 
Performance  Targets 


The  extent  to  which  the  three  selected  MAIS  programs  in  our  study 
experienced  changes  in  their  cost  and  schedule  estimates  and  met 
performance  targets  varied.  Specifically,  the  Army  and  Air  Force 
programs  experienced  slight  changes  in  their  cost  and  schedule 
estimates,  while  the  Navy  program  experienced  more  significant  changes. 
In  addition,  only  one  program,  Air  Force’s  DEAMS,  did  not  fully  meet  its 
technical  performance  targets.  Table  1  provides  a  status  of  the  cost, 
schedule  changes,  and  the  results  of  technical  performance  targets  for 
the  programs.  See  appendix  II  for  the  detailed  profiles  of  each  program. 


25GAO,  Standards  for  Internal  Control  in  the  Federal  Government,  GAO-1 4-704G 
(Washington,  D.C.:  Sept.  10,  2015). 
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Table  1 :  Extent  of  Changes  in  Cost  and  Schedule  Estimates,  and  the  Results  of  Technical  Performance  Targets  (from  the  first 
acquisition  baseline  estimate) 


Component/ 

program 

Cost  increase 

Percentage  change 

Schedule  change 

Met  technical 
performance  targets? 

Army/ 

TMC 

$368  million 

19% 

3  months  (full  deployment) 

✓ 

Navy/ 

CAC2S 

$1 .66  billion 

477% 

13  years  9  months  (full 
deployment) 

✓ 

Air  Force/ 

DEAMS 

$129  million 

9% 

6  months  (milestone  C) 

Source:  GAO  analysis  of  data  provided  by  DOD  officials  |  GAO-16-336 


As  of  January  2016,  the  latest  life-cycle  cost  estimate  for  the  Army’s  TMC 
program  had  increased  about  19  percent  from  the  program’s  February 
2008  acquisition  program  baseline  estimate  (from  approximately  $1.97 
billion  up  to  $2.34  billion).  Program  officials  attributed  the  cost  increase  to 
a  breach  in  the  research  and  development  testing  and  evaluation  cost 
estimation  that  was  reported  to  Congress.  The  program’s  estimated 
program  development  cost  increased  by  45  percent  over  the  original 
acquisition  program  baseline  due  to  program  scope  changes  derived  from 
the  realignment  of  certain  missions,  such  as  the  endorsement  of  the 
Command  Post  of  the  Future  as  a  foundation  for  mission  command. 


Army’s  Program  Had  a 
Slight  Cost  Increase,  but 
Was  Within  Schedule  and 
Met  All  Its  Performance 
Targets 


As  of  January  2016,  TMC  program  experienced  a  3-month  slippage  in  its 
full  deployment  date,  currently  scheduled  for  December  2018.  The 
slippage  was  within  the  program’s  pre-established  threshold  allowance  to 
account  for  minor  changes  in  schedule  and,  program  officials  stated  that 
this  slippage  was  considered  to  be  a  low  risk  that  the  program  has 
accepted.  Program  officials  stated  that,  although  the  Command  Post  of 
the  Future  product  is  95  percent  fielded  and  is  on  schedule  to  reach  full 
deployment  by  December  2018,  continued  support  of  the  Command  Post 
Computing  Environment  is  needed  beyond  fiscal  year  2019. 

As  of  January  2016,  the  TMC  program  met  all  three  of  its  key 
performance  targets,  which  include  supporting  net-centric  military 
operations,  disseminating  orders  with  future  Army  and  Joint  Command 
and  Control  Systems,  and  displaying  unified  information  on  subject 
matters. 
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The  Navy  Program 
Experienced  Significant 
Cost  and  Schedule 
Increases,  but  Met  All 
Performance  Targets 


As  of  October  2015,  the  latest  life-cycle  cost  estimate  for  Navy’s  CAC2S 
Increment  1  program  had  increased  about  477  percent  from  its  first 
acquisition  program  baseline  estimate  (from  approximately  $347  million 
up  to  $2  billion).  As  previously  reported,26  factors  attributed  to  the 
program’s  early  developmental  challenges  contributed  to  an  increase  in 
its  cost  estimate,  which  include  program  scope  growth  and  restructuring. 
According  to  program  documentation,  despite  the  program’s  initial 
challenges  in  its  cost  estimates  due  to  an  increase  in  its  operations  and 
support  expenditures,  the  CAC2S  program  has  demonstrated  gradual 
improvement  as  it  reported  a  cost  avoidance  of  $54.4  million  for 
implementing  the  DOD  Better  Buying  Power  initiatives  that  befitted  from 
competitive  market  forces  that  drove  down  cost.  As  of  October  2015,  the 
program’s  latest  life-cycle  estimate  relative  to  its  November  2010 
production  acquisition  program  baseline  cost  estimate  had  decreased  by 
about  19  percent  (from  approximately  $2.46  billion  down  to  $2  billion). 

As  of  October  2015,  compared  to  its  first  acquisition  program  baseline 
schedule,  the  program  experienced  a  13  year  and  9  month  slippage  in  its 
full  deployment  date — currently  scheduled  for  March  2022.  As  previously 
reported,27  factors  that  attributed  to  the  prior  schedule  slippage  included 
the  addition  of  new  requirements  and  program  restructuring.  However, 
program  officials  stated  that  the  program  has  been  executing  in 
accordance  within  its  approved  schedule.  As  of  October  2015,  the 
estimated  milestone  C  phase  2  was  delayed  by  6  months  from  the 
program’s  production  acquisition  program  baseline  but  achieved  it 
milestone  approved  within  the  program’s  pre-established  schedule 
threshold.  Program  officials  attributed  this  delay,  in  part,  to  administrative 
factors,  which  included  the  review  and  approval  process.  CAC2S 
successfully  achieved  milestone  C  phase  2  approval  in  February  2015  but 
the  acquisition  decision  memorandum  had  not  been  signed  until  March 
2015. 


As  of  October  2015,  program  officials  reported  that,  during  performance 
testing,  it  was  meeting  both  of  its  key  performance  targets  related  to  net- 
ready  and  data  fusion. 


26GAO-1 3-311. 
27GAO-1 3-311. 


Page  18 


GAO-16-336  Major  Automated  Information  Systems 


Air  Force’s  Program  Had 
Slight  Cost  and  Schedule 
Increases  and  Did  Not 
Meet  Performance  Targets 


As  of  October  2015,  the  latest  life-cycle  cost  estimate  for  the  Air  Force’s 
DEAMS  program  had  increased  about  9  percent  from  its  first  February 
2012  acquisition  program  baseline  estimate  (from  approximately  $1.43 
billion  up  to  $1.56  billion).  Program  officials  attributed  the  cost  increase,  in 
part,  to  program  scope  growth  and  the  addition  of  software  upgrade 
enhancements.  Specifically,  as  of  October  201 5,  the  program’s  life-cycle 
cost  estimate  incorporated  additional  infrastructure  maintenance  costs 
throughout  the  life-cycle  that  added  performance  monitoring  and 
additional  deployment  support.  Also,  according  to  program  officials,  the 
program  brought  forward  increment  2  requirements  and  a  second  Oracle 
software  upgrade  in  year  2021 . 


DEAMS  experienced  a  6  month  slip  in  its  milestone  C  but  was  within  its 
threshold,  a  predefined  point  where  programs  that  exceed  it  are  at 
increased  risk.  However,  it  did  experience  a  1  year  slip  in  its  full 
deployment  decision  date — currently  scheduled  for  February  2016. 
Program  officials  attributed  this  slippage  due  to  findings  identified 
DEAMS’s  initial  operational  test  and  evaluation  report.  While  the  program 
had  been  established  since  August  2003,  a  full  deployment  date  had  not 
been  determined.  As  of  September  2015,  program  officials  expect  full 
deployment  to  be  reached  by  October  2016. 

As  of  October  2015,  DEAMS  program  officials  reported  that  the  program 
did  not  meet  all  of  its  nine  key  performance  targets.  Specifically,  DEAMS 
did  not  meet  five  performance  targets:  Balance  with  Treasury,  Accurate 
Balance  of  Available  Funds,  Timely  Reporting,  Period-End  Processing, 
and  Net-Ready.  For  example,  the  two  operational  assessments  that  were 
conducted  from  2012  to  201428  identified  significant  weaknesses29  in 
three  measures  of  effectiveness  and  suitability.  Further,  the  Initial 
Operational  Test  and  Evaluation30  report  identified  system  performance 
issues  within  the  DEAMS  program,  which  included  change  management 
issues,  transaction  backlogs,  and  ineffective  reporting  tools. 


oo 

The  first  operational  assessment  was  conducted  by  the  Air  Force  Test  and  Evaluation 
Center  from  May  2012  to  until  June  2012.  The  second  operational  assessment  was 
conducted  by  the  Air  Force  Test  and  Evaluation  Center  and  the  Office  of  the  Director, 
Operational  Test  and  Evaluation  from  September  2013  to  January  2014. 

oq 

A  significant  weakness  is  defined  as  unlikely  to  meet  criteria — significant  shortfall  or 
unlikely  to  meet  identified  standard — significant  shortfall. 

30lnitial  Operational  Test  and  Evaluation  was  conducted  by  the  Air  Force  Operational  Test 
and  Evaluation  Center  from  October  2014  to  May  2015. 
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Subsequently,  the  Air  Force  Operational  Test  and  Evaluation  Center 
provided  29  recommendations  for  the  Air  Force  to  implement  to  support 
the  successful  fielding  of  DEAMS  Increment  1,17  of  which  were 
documented  as  being  completed,  while  corrective  action  for  the  remaining 
12  are  still  underway.  However,  according  to  program  documentation, 
DEAMS  must  demonstrate  measureable  improvement  by  the  full 
deployment  decision  date  of  February  2016,  in  order  to  avert  future 
schedule  delays  in  its  fielding  deployment. 


Selected  Programs 
Use  of  IT  Acquisition 
Best  Practices  Were 
Mostly  Applied  for 
Requirements 
Management  and 
Fully  Applied  for  Risk 
Management 


According  to  the  Software  Engineering  Institute’s  Capability  Maturity 
Model  Integration  for  Acquisition,  an  appropriate  requirements 
management  involves  establishing  an  agreed-upon  set  of  requirements, 
ensuring  traceability  between  requirements  and  work  products,  and 
managing  any  changes  to  the  requirements  in  collaboration  with 
stakeholders.  Likewise,  an  effective  risk  management  process  identifies 
potential  problems  before  they  occur,  so  that  risk-handling  activities  may 
be  planned  and  invoked,  as  needed,  across  the  life  of  the  project  in  order 
to  mitigate  the  potential  for  adverse  impacts.  Table  2  provides  key 
practices  used  to  comprehensively  manage  requirements  and  risk. 


Table  2:  Key  Requirements  and  Risk  Management  Best  Practices 

Requirements  management  practices 

Risk  management  practices 

Develop  an  understanding  with  the  requirements  providers  on  the 
meaning  of  the  requirements. 

Determine  risk  sources  and  categories  used  to  examine  and 
oversee  changes  that  impact  the  project. 

Obtain  commitment  to  requirements  from  project  participants. 

Define  parameters  used  to  analyze  and  categorize  risks  and  to 
control  the  risk  management  effort. 

Manage  changes  to  requirements  as  they  evolve  during  the 
project. 

Establish  and  maintain  the  strategy  to  be  used  for  risk 
management. 

Maintain  bidirectional  traceability  among  requirements  and  work 
products. 

Identify  and  document  risks  that  could  negatively  affect  work 
efforts. 

Ensure  that  project  plans  and  work  products  remain  aligned  with 
requirements. 

Evaluate  and  categorize  each  identified  risk  using  defined  risk 
categories  and  parameters,  and  determine  its  relative  priority. 

Develop  a  risk  mitigation  plan  in  accordance  with  the  risk 
management  strategy. 

Monitor  the  status  of  each  risk  periodically  and  implement  the  risk 
mitigation  plan  as  appropriate. 

Source:  GAO  analysis.  |  GAO-16-336 


All  three  selected  programs  implemented  IT  acquisition  best  practices  for 
risk  management,  but  requirements  management  best  practices  were  not 
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consistently  implemented  by  the  programs.  Table  3  provides  a  summary 
of  the  extent  to  which  requirements  and  risk  management  best  practices 
were  implemented  by  each  program. 


Table  3:  Extent  to  Which  Best  Practices  Were  Implemented  by  Program 

Component/ 

program 

Requirements  management 
practices 

Risk  management 
practices 

Army / 

TMC 

3  of  5 

7  of  7 

Navy/ 

CAC2S 

3  of  5 

7  of  7 

Air  Force/ 

DEAMS 

4  of  5 

7  of  7 

Source:  GAO  analysis.  |  GAO-16-336 


The  Army  Program 
Implemented  Risk 
Management  Practices, 
but  Did  Not  Apply  Several 
Requirements 
Management  Practices 


The  Army  implemented  all  risk  management  best  practices  for  the  TMC 
program.  For  example,  the  risk  management  plan,  dated  May  2014, 
identified  risk  sources  to  include,  among  other  things,  unclear  system 
requirements,  immature  technology,  and  an  unstable  organizational 
environment.  In  addition,  program  officials  analyzed,  categorized,  and 
controlled  risks  using  a  probability  and  impact  model  that  considered  the 
risks  (from  very  low  to  very  high)  and  the  potential  consequences.  The 
program  also  used  a  risk  radar  tool  to  track  and  monitor  risks.  These  are 
reviewed  weekly  during  staff  meetings  and  updated  monthly.  Further, 
TMC’s  risk  management  plan  indicated  that  contingency  plans  are 
invoked  whenever  adjustments  to  cost,  schedule,  or  performance  are 
required.  In  taking  these  and  other  actions,  the  TMC  program  had 
established  and  utilized  the  key  risk  management  practices.  Doing  so 
should  better  position  the  program  to  mitigate  adverse  impacts  from 
potential  problems  before  they  occur. 


The  Army  had  implemented  three  requirements  management  best 
practices  for  the  program,  but  did  not  fully  implement  two:  the  practice  of 
managing  requirements  changes  and  ensuring  that  work  products  are  in 
alignment  with  requirements.  For  example,  one  key  practice  that  the 
program  implemented  included  the  maintenance  of  the  bidirectional 
traceability  tool  among  requirement.  Specifically,  program  officials  utilized 
a  traceability  tool  used  to  generate  a  requirements  matrix  to  track  all  of  its 
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program  elements  to  the  requirements.  Regarding  managing 
requirements  changes,  while  program  officials  tracked  requirements 
changes  in  a  database,  requirements  changes  were  not  always  available 
at  the  stakeholder  level  to  evaluate  the  impact  and  determine  the  status 
of  requirements  changes  for  all  elements  of  the  program.  Even  though 
TMC  was  in  the  production  phase,  it  relied  solely  on  the  functions  within 
its  requirements  database  rather  than  a  requirements  management 
document.  This  left  the  program  without  a  formal  mechanism  to  track  and 
ensure  project  plans,  activities,  and  work  products  are  consistent  with 
defined  requirements.  Without  such  a  document,  the  program  does  not 
have  a  formal  mechanism  to  track  and  ensure  that  project  plans, 
activities,  and  work  products  are  consistent  with  defined  requirements. 


The  Navy  Program 
Implemented  Risk 
Management  Practices, 
but  Did  Not  Apply  All 
Requirements 
Management  Practices 


The  Navy  implemented  all  risk  management  best  practices  for  the  CAC2S 
program.  For  example,  the  risk  management  plan  assessed  risks  in  terms 
of  their  probability  and  consequence  of  occurrence.  The  program  also 
identified  and  documented  risks  and  had  the  supporting  documentation 
that  included  risk  and  issue  register  logs,  detailed  reports,  the  integrated 
master  schedules,  and  risk  assessments.  In  addition,  the  risk 
management  plan  established  the  strategy  that  included  the  processes  to 
guide  risk  mitigation  efforts  at  the  lowest,  appropriate  level.  The 
program’s  risk  registers,  which  included  risk  mitigation  steps,  were 
provided  by  program  officials  to  demonstrate  that  risk  mitigation  plans  had 
been  implemented  for  each  risk.  In  taking  these  and  other  actions,  the 
program  had  established  and  utilized  effective  risk  management 
practices. 

The  Navy  implemented  three  requirements  management  best  practices 
for  the  program  but  did  not  implement  two.  For  example,  one  key  practice 
implemented  was  managing  requirements  changes.  Specifically,  the 
program  office  demonstrated  the  ability  to  effectively  manage  changes  to 
requirements  as  they  evolved  during  the  project.  Program  officials  did  this 
by  documenting  the  alignment  of  requirements  to  its  respective 
requirements  changes,  maintaining  a  history  of  requirements  changes 
with  rationale  explaining  the  change  request  within  its  configuration 
documentation,  and  publishing  requirements  data  using  a  database  tool. 
However,  the  program  did  not  fully  implement  the  practice  of  maintaining 
traceability  among  requirements  and  work  products  to  ensure  that  work 
products  were  in  alignment. 

Further,  with  regard  to  the  program’s  bidirectional  traceability  tool, 
according  to  the  July  2014  traceability  tool,  25  specifications  and  13 
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capabilities  did  not  map  to  its  respective  work  products.  According  to 
program  officials,  as  of  November  2015,  1 1  of  the  25  specifications  had 
been  mapped,  but  due  to  an  oversight,  mapping  of  these  specifications 
were  not  associated,  while  the  remaining  14  specifications  did  not  map  to 
its  respective  capability  production  documentation  to  demonstrate 
completeness.  According  to  program  officials,  4  capabilities  had  been 
recently  mapped  but  the  remaining  9  capabilities  listed  in  the  capability 
production  documentation  did  not  map  the  traceability  to  the  respective 
requirements.  According  to  program  officials,  the  mapping  discrepancy  of 
7  capabilities  was  attributed  to  unfunded,  obsolete,  and  programmatic 
requirements.  Furthermore,  after  notifying  the  program  of  the  gaps  we 
identified,  officials  stated  that  they  would  take  action  to  ensure  mapping  of 
14  specifications  and  2  capabilities  to  their  respective  requirements  work 
products  would  be  addressed. 

Regarding  the  alignment  of  requirements,  the  program’s  requirements 
management  plan  had  not  been  updated  since  May  2009  and  software 
specifications  and  capabilities  were  not  consistently  maintained. 

According  to  program  officials,  the  current  requirements  management 
plan  had  been  previously  reviewed  and  was  determined  to  be  suitable  for 
the  purpose  of  implementing  requirements  management  best  practices. 
According  to  the  Capability  Maturity  Model®  Integration  for  Acquisition, 
without  a  clear  linkage  between  requirements  all  the  lower-level 
requirements  and  capabilities,  the  program  may  not  be  effectively 
managing  development  efforts  in  accordance  with  the  most  recent 
requirements.  Until  work  products  are  updated,  the  program  cannot 
provide  assurance  that  its  requirements  are  aligned  with  the  most 
updated  work  products  and  is  at-risk  of  potential  cost  and  schedule 
consequences. 


The  Air  Force  Program 
Implemented  Risk 
Management  Practices, 
but  Did  Not  Apply  One 
Requirements 
Management  Practice 


The  Air  Force  implemented  all  seven  risk  management  best  practices  for 
the  DEAMS  program.  To  the  program’s  credit,  DEAMS  demonstrated 
great  strides  in  improving  its  risk  management  best  practices.  Since  our 
prior  MAIS  review,31  the  program  has  made  improvements  such  as 
monitoring  the  status  of  each  risk  periodically,  and  ensuring  that  risk 
reports  were  up  to  date,  which  included  the  status  of  actions  to  mitigate 
risks.  Other  key  practices  include  defining  parameters  to  analyze  and 


31 GAO-1 3-311. 
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categorize  risks,  documenting  risk,  and  developing  risk  mitigation  plans  in 
accordance  with  the  risk  management  strategy,  among  other  areas.  In 
taking  these  and  other  actions,  the  DEAMS  program  had  established  and 
utilized  effective  risk  management  practices. 

The  Air  Force  had  implemented  four  requirements  management  best 
practices,  but  did  not  fully  implement  one  requirements  management 
practice,  the  practice  of  developing  an  understanding  with  providers  on 
the  meaning  of  requirements.  For  example,  the  Air  Force  ensured  that 
project  plans  and  work  products  were  aligned  with  the  most  recent 
requirements.  Specifically,  the  program  maintained  consistent 
documentation  and  oversight  of  work  products,  which  included  an  up-to- 
date  requirements  management  plan,  system  specifications,  and 
capabilities  documentation.  However,  while  the  program  had  established 
an  adjudication  process  by  which  requirements  were  reviewed  and 
approved,  and  implemented  a  test  methodology  to  validate  requirements 
prior  to  production  installation,  the  function  to  ensure  accountability  was 
not  working  properly. 

As  such,  program  officials  did  not  determine  whether  key  requirements 
were  validated  during  system  integration  testing  prior  to  deploying 
software  into  production,  which  was  released  with  unresolved  issues. 
Program  officials  subsequently  resolved  the  issues  without  any  negative 
impacts.  The  program  office  attributed  the  issue  of  not  fully  validating 
requirements  to  environmental  issues  that  were  considered  to  be 
acceptable  risks  and,  subsequently,  scheduled  production  installation  in 
November  2014.  Nevertheless,  according  to  the  Software  Engineering 
Institute’s  Capability  Maturity  Model®  Integration  for  Acquisition, 
requirements  should  be  analyzed  to  ensure  that  established  criteria  are 
met  so  that  proper  control  functions  are  in  place. 
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All  Appropriate  MAIS 
Programs  Were 
Represented  on  the 
Federal  IT 
Dashboard,  but  the 
Responsible 
Organization  Was  Not 
Provided 


Pursuant  to  its  statutory  responsibility  to  analyze,  track,  and  evaluate 
risks,  OMB  requires  agency  CIOs  to  provide  cost,  schedule,  and  risk 
information  for  all  major  IT  investments  on  the  IT  Dashboard.32  In 
addition,  the  IT  Dashboard  shows  CIO  names  and  photographs  who  are 
responsible  for  investments  to  increase  accountability  for  IT  acquisitions. 

As  of  October  2015,  27  of  the  39  DOD  MAIS  programs  were  listed  on  the 
IT  Dashboard.  According  to  DOD  officials,  and  in  accordance  with  OMB 
policy,  8  MAIS  programs  that  have  not  been  funded  in  the  President’s 
budget  submission  are  not  reported  to  the  Dashboard,  as  appropriate.  In 
addition,  4  MAIS  programs  have  been  designated  by  DOD  as  containing 
national  security-sensitive  information  and  were  therefore  classified  and 
not  subject  to  being  reported  on  the  Dashboard.  According  to  DOD  CIO 
and  AT&L  officials,  the  8  unfunded  MAIS  programs  will  be  reported  to  the 
Dashboard  after  the  President’s  2017  budget  submission  has  been 
finalized. 


However,  the  organization  responsible  for  supervising  MAIS  acquisition 
programs — AT&L — is  not  represented  on  the  Dashboard.  Instead,  the 
Dashboard  publicly  shows  DOD’s  CIO  as  the  responsible  party,  pursuant 
to  OMB’s  direction,  but  is  not  accurate.  AT&L  has  oversight  responsibility 
for  the  acquisition  performance  of  MAIS  programs.  In  this  regard,  not  only 
does  AT&L  supervise  department  acquisitions  and  establish  its 
acquisition  policies,  but  as  the  milestone  decision  authority  for  MAIS 
programs,  the  Linder  Secretary  or  his  designee,  has  overall  responsibility 
for  each  program.  By  contrast,  the  CIO  is  not  involved  in  managing  the 
performance  of  the  MAIS  programs  but  is  responsible  for  submitting  the 
rating  to  the  Dashboard.33 

Officials  from  DOD’s  Office  of  the  CIO  and  OMB’s  Office  of  E- 
Government  and  Information  Technology  told  us  that  they  were  aware  of 
this  inconsistency  on  the  Dashboard  but  did  not  think  it  was  a  significant 
issue.  Further,  the  DOD  officials  stated  that  since  the  CIO  is  involved  in 
the  rating  process  the  representation  of  their  office  on  the  Dashboard  is 
sufficient.  Nonetheless,  since  only  the  DOD  CIO  is  represented  on  the 
Dashboard,  the  public  and  other  users  may  be  unaware  that  AT&L  has 


3240  U.S.C.  §  11302(c). 

33DOD-5000.02,  Enclosure  1  says  the  milestone  decision  authority  for  MAIS  programs 
can  be  delegated  to  a  component  head  or  a  component  acquisition  executive. 
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overall  oversight  for  the  acquisition  performance  of  MAIS  programs, 
minimizing  the  intended  accountability  the  Dashboard  is  to  provide. 


Conclusions 


Since  MAIS  programs  account  for  billions  of  dollars  of  DOD’s  IT  budget,  it 
is  important  that  the  required  critical  change  reports  are  timely  so 
Congress  has  the  necessary  information  to  make  budgetary  and 
oversight  decisions.  While  the  reports  contained  the  required  elements, 
many  were  not  submitted  in  a  timely  manner,  potentially  hampering 
Congress’  ability  to  make  informed  decisions. 

Further,  all  three  selected  programs  implemented  IT  acquisition  best 
practices  for  risk  management  and  implemented  most  practices  for 
requirements  management.  While  this  is  a  significant  achievement, 
improvements  can  be  made  in  managing  requirements.  Among  other 
things,  programs  were  operating  without  a  current  requirements 
management  plan  that  was  considered  to  be  acceptable  risks.  Managing 
requirements  effectively  is  especially  necessary  since  MAIS  programs  are 
intended  to  help  the  department  sustain  its  key  operations. 

Finally,  there  is  a  lack  of  accountability  for  AT&L  on  the  IT  Dashboard. 
While  OMB  intended  for  accountability  by  requiring  that  major 
investments  show  agency  CIOs  as  responsible,  it  did  not  consider  that 
DOD’s  AT&L  is  the  responsible  party  for  oversight  of  the  acquisition 
performance  of  MAIS  programs.  Since  the  Dashboard  does  not  reflect 
that  AT&L  has  such  responsibility,  there  is  decreased  public 
accountability. 


Rprnmmpnrlatinnq  fnr  'mProve  the  management  of  MAIS  programs,  we  are  making  six 

recommendations  that: 

Executive  Action 

•  The  Secretary  of  Defense  examine  the  MAIS  critical  change  reporting 
process  to  identify  root  causes  for  delays  and  implement  corrective 
actions  for  the  timely  delivery  of  critical  change  reports. 

•  The  Secretary  of  Defense  develop  a  mechanism  for  monitoring 
whether  MAIS  programs  with  late  reports  are  restricted  from  obligating 
funds  and  in  turn  ensuring  compliance  with  the  Antideficiency  Act. 

•  The  Secretary  of  the  Army  direct  the  TMC  program  manager  to 
develop  a  requirements  management  plan  to  document  and  manage 
its  requirements  process. 

•  The  Secretary  of  the  Navy  direct  the  CAC2S  program  manager  to 
identify  weaknesses  in  the  requirements  traceability  process  and  take 
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corrective  actions  to  manage  the  traceability  of  requirements  to  the 
respective  lower-level  requirements,  and  periodically  evaluate  work 
products,  including  the  requirements  management  plan,  and  update 
them  in  accordance  with  the  requirements  guidance. 

•  The  Secretary  of  the  Air  Force  direct  the  DEAMS  program  manager  to 
address  weaknesses  in  its  controls  for  ensuring  that  all  software 
requirements  are  tested  and  validated  before  deployment  of  new 
software  releases. 

•  Director  of  OMB  instruct  the  Federal  CIO  to  add  the  Under  Secretary 
of  Defense  for  AT&L  as  a  responsible  party  to  DOD’s  MAIS  entries  on 
the  Federal  IT  Dashboard  website,  alongside  the  CIO,  to  publicly 
disclose  the  responsible  party  for  the  acquisition  performance 
management  of  MAIS  programs. 


Agency  Comments 
and  Our  Evaluation 


We  provided  a  draft  of  this  report  to  DOD  and  OMB.  We  received  written 
comments  from  DOD’s  Acting  Principal  Deputy  Assistant  Secretary  of 
Defense  for  Acquisition,  which  are  reprinted  in  appendix  III.  In  its 
comments,  the  department  concurred  with  all  five  recommendations  to 
improve  oversight,  IT  acquisition  practices,  and  tools  used  to  manage 
MAIS  programs. 

In  e-mail  comments,  an  official  from  OMB’s  audit  liaison  group  stated  that 
OMB’s  Office  of  E-Government  and  Information  Technology  does  not 
agree  with  the  recommendation  to  add  AT&L  to  the  IT  Dashboard  as  a 
responsible  party  for  MAIS  programs  but  would  work  with  DOD  to 
address  it.  The  official  did  not  provide  a  rationale  for  this  position  or 
explain  how  OMB  would  work  with  DOD. 

Nonetheless,  we  continue  to  believe  there  is  a  lack  of  transparency  and 
accountability  for  AT&L  on  the  IT  Dashboard.  The  IT  Dashboard  publicly 
shows  DOD’s  CIO  as  the  responsible  party,  pursuant  to  OMB’s  direction. 
However,  AT&L  has  oversight  responsibility  for  the  acquisition 
performance  of  MAIS  programs.  In  this  regard,  not  only  does  AT&L 
supervise  department  acquisitions  and  establish  its  acquisition  policies, 
but  as  the  milestone  decision  authority  for  MAIS  programs,  the  Under 
Secretary  or  his  designee,  has  overall  responsibility  for  each  program. 
The  DOD  CIO  is  not  involved  in  managing  the  performance  of  the 
programs  and  is  only  responsible  for  submitting  the  rating  to  the 
Dashboard.  We  believe  that  adding  AT&L  to  the  IT  Dashboard  would 
increase  public  accountability  and  leadership  transparency  for  the 
acquisition  management  of  MAIS  programs. 
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We  are  sending  copies  of  this  report  to  the  appropriate  congressional 
committees;  the  Secretary  of  Defense;  the  Secretary  of  the  Air  Force,  the 
Secretary  of  the  Army,  the  Secretary  of  the  Navy,  the  Office  of 
Management  and  Budget,  and  other  interested  parties.  This  report  also  is 
available  at  no  charge  on  the  GAO  website  at  http://www.gao.gov. 

Should  you  or  your  staffs  have  any  questions  on  information  discussed  in 
this  report,  please  contact  me  at  (202)  512-4456  or  ChaC@gao.gov. 
Contact  points  for  our  Offices  of  Congressional  Relations  and  Public 
Affairs  may  be  found  on  the  last  page  of  this  report.  GAO  staff  who  made 
major  contributions  to  this  report  are  listed  in  appendix  IV. 


Carol  R.  Cha 
Director 

Information  Technology  Acquisition  Management  Issues 
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List  of  Committees 


The  Honorable  John  McCain 
Chairman 

The  Honorable  Jack  Reed 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Ron  Johnson 
Chairman 

The  Honorable  Thomas  R.  Carper 
Ranking  Member 

Committee  on  Homeland  Security  and  Governmental  Affairs 
United  States  Senate 

The  Honorable  Thad  Cochran 
Chairman 

The  Honorable  Richard  J.  Durbin 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Mac  Thornberry 
Chairman 

The  Honorable  Adam  Smith 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  Jason  Chaffetz 
Chairman 

The  Honorable  Elijah  Cummings 
Ranking  Member 

Committee  on  Oversight  and  Government  Reform 
House  of  Representatives 
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The  Honorable  Rodney  Frelinghuysen 
Chairman 

The  Honorable  Pete  Visclosky 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 
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Appendix  I:  Objectives,  Scope,  and 
Methodology 


The  National  Defense  Authorization  Act  for  Fiscal  Year  2012  includes  a 
provision  that  we  select,  assess,  and  report  on  selected  Department  of 
Defense  (DOD)  major  automated  information  system  (MAIS)  programs 
annually  through  March  201 8. 1  In  addition,  Senate  Report  113-176 
accompanying  S.  2410  includes  a  provision  that  we  evaluate  DOD’s 
implementation  of  statutory  reporting  requirements  for  MAIS  programs 
experiencing  a  critical  change.  Our  objectives  for  this  report  were  to  (1 ) 
evaluate  DOD’s  implementation  of  statutory  reporting  requirements  for 
MAIS  programs  experiencing  a  critical  change;  (2)  describe  the  extent  to 
which  selected  MAIS  programs  have  changed  their  planned  cost  and 
schedule  estimates,  and  met  performance  targets;  (3)  assess  the  extent 
to  which  selected  MAIS  programs  have  used  key  IT  acquisition  best 
practices,  including  risk  management;  and  (4)  determine  the  extent  to 
which  MAIS  programs  are  accurately  represented  on  the  Federal  IT 
Dashboard. 

To  evaluate  DOD’s  implementation  of  statutory  reporting  requirements  for 
MAIS  programs  experiencing  a  critical  change — a  schedule  delay  of  1 
year  or  more,  a  full  life-cycle  cost  increase  of  25  percent  or  more  over  the 
original  estimate,  or  a  change  that  will  undermine  the  system’s  ability  to 
perform  as  intended — we  collected  and  analyzed  information  about 
critical  changes  from  December  2008  to  June  2014  and  their 
corresponding  reports.  We  assessed  whether  DOD  had  met  reporting 
requirements  by  reviewing  MAIS  critical  change  reports  and  supporting 
documentation  to  determine  if  they  included  required  written  certifications 
stating  that: 

•  the  automated  information  system  or  IT  investment  to  be  acquired  is 
essential  to  the  national  security  or  to  the  efficient  management  of  the 
DOD; 

•  there  is  no  alternative  to  the  system  or  IT  investment  which  will 
provide  equal  or  greater  capability  at  less  cost; 

•  the  new  estimates  of  the  costs,  schedule,  and  performance 
parameters  have  been  determined,  with  the  concurrence  of  the 
Director  of  Cost  Assessment  and  Program  Evaluation,  to  be 
reasonable;  and 

•  the  management  structure  for  the  program  is  adequate  to  manage 
and  control  program  costs. 


Vub.  L.  No.  112-81,  §  1078  (Dec.  31, 2011).  Under  this  provision  we  report  on  these 
assessments  no  later  than  March  30  of  each  year  from  2013  through  2018. 
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Methodology 


We  did  not  look  at  the  quality  of  the  assessments  and  estimate  done.  We 
reviewed  the  reports  to  determine  if  the  particular  elements  were 
included.  We  then  summarized  the  number  of  elements  addressed  by 
program. 

In  order  to  determine  if  the  critical  change  reports  met  or  exceeded  the 
60-day  requirement,  we  collected  all  MAIS  reports  and  compared  the 
dates  within  them.  We  subtracted  the  date  in  which  the  reports  were 
delivered  to  Congress  via  letter  from  the  date  in  which  the  program 
manager  provided  the  quarterly  report  to  the  senior  DOD  official 
responsible  for  the  program  in  order  to  determine  the  number  of  days  that 
had  lapsed.  Additionally,  we  rearranged  the  programs  by  number  of  days 
(from  greatest  to  least)  it  took  to  deliver  the  critical  change  reports.  We 
noted  how  many  programs  took  more  than  100  days,  and  200  days  to 
deliver  the  reports.  We  also  interviewed  DOD  officials  in  order  to  ask  their 
opinion  on  the  length  of  time  it  takes  to  deliver  a  critical  change  report  to 
Congress. 

We  interviewed  DOD  officials  responsible  for  the  quality  of  the  data  and 
assessed  their  procedures  for  maintaining  its  accuracy  and 
completeness.  In  addition,  we  examined  the  data  for  outliers  or  others 
extraordinary  items.  Based  on  these  procedures,  we  have  concluded  that 
these  data  are  sufficiently  reliable  for  our  purposes. 

Additionally,  in  order  to  determine  whether  DOD  is  tracking  the  obligation 
of  funds  during  a  program’s  critical  change,  we  interviewed  DOD  officials 
in  order  to  determine  what  processes  and  tools  they  had  in  place  to 
ensure  that  funds  were  not  being  obligated. 

To  address  the  second  and  third  objectives,  we  used  DOD’s  official  list  of 
39  MAIS  programs,  as  of  February  25,  2015,  to  establish  the  basis  for 
selecting  the  MAIS  programs  that  were  used  to  assess  objectives  two  and 
three.  We  used  the  criteria  below  to  select  three  MAIS  programs. 

Any  programs  that  were  used  in  the  prior  two  reviews  should  be 
excluded. 

•  Any  programs  that  are  fully  deployed  or  cancelled  should  be 
eliminated  from  consideration. 

•  The  program  should  not  be  new  to  the  MAIS  list;  otherwise,  there  may 
not  be  sufficient  acquisition  activity  and  documentation  to  evaluate. 

•  The  program  must  have  a  baseline  in  order  to  have  a  reference  point 
for  evaluating  cost  and  schedule  performance  characteristics. 
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The  program  cannot  be  a  National  Security  Agency  program. 

•  We  included  one  program  from  each  military  department — Army,  Air 
Force,  and  Navy  in  order  to  diversify  the  portfolio.  Thus,  we  excluded 
any  DOD-wide  programs. 

•  We  selected  programs  that  had  the  lowest  relative  ratings  on  the 
Federal  IT  Dashboard  as  of  April  201 5. 2 

•  We  preferred  that  the  program  be  complex  (e.g.,  integration  across 
domains,  global,  critical  to  battle  operations,  etc.),  rather  than  an 
upgrade. 

•  We  preferred  to  select  programs  with  funding  profiles  that  are 
significant  when  compared  to  the  rest  of  the  portfolio. 

•  We  considered  issues  identified  from  credible  sources  of  information, 
such  as  Defense  Acquisition  Management  Information  Retrieval 
online  resources,  IT  Dashboard  ratings,  etc. 

We  filtered  the  original  list  of  MAIS  programs  using  the  criteria  above. 
Based  on  this  filtering,  we  chose  the  following  systems: 

•  the  Air  Force’s  Defense  Enterprise  Accounting  and  Management 
System-Increment  1  (DEAMS  Increment  1), 

•  the  Army’s  Tactical  Mission  Command  (TMC),  and 

•  the  Navy’s  Common  Aviation  Command  and  Control  System 
Increment  1  (CAC2S  Increment  1). 

To  address  the  second  objective,  we  analyzed  and  compared  each 
selected  program’s  first  acquisition  program  baseline  cost  estimate  to  the 
latest  life-cycle  estimate  to  determine  the  extent  to  which  planned 
program  costs  had  changed.  Similarly,  to  determine  the  extent  to  which 
these  programs  changed  their  planned  schedule  estimates,  we  compared 
each  program’s  first  acquisition  program  baseline  schedule  to  the  latest 
schedule.  We  relied  on  the  thresholds  established  by  statute  to  describe 
the  amount  of  any  deviation  (i.e.,  significant  or  critical)  that  each 
program’s  latest  life-cycle  cost  and  schedule  estimates  experienced  from 
the  first  acquisition  program  baseline. 


2DOD  had  no  MAIS  programs  listed  as  moderately  high  risk  -  “2”  -  or  high  risk  -  “1  ”  -  so 
we  chose  programs  from  the  Army  and  Navy  that  were  rated  as  medium  risk  -  “3.”  The  Air 
Force  had  no  programs  rated  as  a  “3”  on  the  Federal  IT  Dashboard,  so  we  chose  an  Air 
Force  program  that  was  rated  as  a  “4.” 
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To  determine  whether  the  selected  programs  met  their  performance 
targets,  we  compared  program  and  system  performance  targets  against 
actual  performance  data  in  test  reports  and  program  management 
briefings.  We  reviewed  the  results  of  operational  assessments  and 
program  evaluations  conducted  on  the  systems.  We  also  reviewed 
additional  information  on  each  program’s  cost,  schedule,  and 
performance,  including  program  documentation,  such  as  DOD’s  MAIS 
annual  and  quarterly  reports;  information  from  the  Office  of  Management 
and  Budget’s  (OMB)  IT  Dashboard;3  acquisition  program  baselines; 
monthly  status  briefings;  system  test  reports;  and  our  prior  reports.  We 
also  interviewed  program  officials  from  each  of  the  selected  MAIS 
programs  to  obtain  additional  information  on  cost,  schedule,  and 
performance.  We  provided  our  assessments  to  the  program  management 
offices  of  each  selected  program  for  comment.  We  aggregated  and 
summarized  the  results  of  these  analyses  across  the  programs,  as  well 
as  developed  individual  profiles  for  each  program  (see  appendix  II). 

To  address  the  third  objective,  we  analyzed  each  selected  program’s  IT 
acquisition  documentation  and  compared  it  to  key  requirements 
management  and  risk  management  best  practices — including  Software 
Engineering  Institute’s  Capability  Maturity  Model®  Integration  for 
Acquisition  (CMMI-  ACQ)4  practices — to  determine  the  extent  to  which 
the  programs  were  implementing  these  practices.  In  particular,  the  key 
requirements  management  best  practices  we  reviewed  were: 

•  develop  an  understanding  with  the  requirements  providers  on  the 
meaning  of  the  requirements, 

•  obtain  commitment  to  requirements  from  project  participants, 

•  manage  changes  to  requirements  as  they  evolve  during  the  project, 

•  maintain  bidirectional  traceability  among  requirements  and  work,  and 

•  ensure  that  project  plans  and  work  products  remain  aligned  with 
requirements. 

Specifically,  we  analyzed  program  requirements  documentation,  including 
requirements  management  plans,  requirements  traceability  matrices, 


3OMB’s  IT  Dashboard  is  a  public  website  that  provides  detailed  information  on  federal 
agencies’  major  IT  investments,  including  assessments  of  actual  performance  against  cost 
and  schedule  targets. 

4Software  Engineering  Institute,  Capability  Maturity  Model®  Integration  for  Acquisition 
(CMMI-ACQ),  Version  1 .3  (November  2010). 
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requirements  change  forms,  technical  performance  assessments,  and 
requirements  board  meeting  minutes.  Additionally,  we  interviewed 
program  officials  to  obtain  additional  information  about  their  requirements 
management  practices. 

Additionally,  we  reviewed  the  following  key  risk  management  best 
practices: 

•  determine  risk  sources  and  categories; 

•  define  parameters  used  to  analyze  and  categorize  risks  and  to  control 
the  risk  management  effort; 

•  establish  and  maintain  the  strategy  to  be  used  for  risk  management; 

•  identify  and  document  risks; 

•  evaluate  and  categorize  each  identified  risk  using  defined  risk 
categories  and  parameters,  and  determine  its  relative  priority; 

•  develop  a  risk  mitigation  plan  in  accordance  with  the  risk  management 
strategy;  and 

•  monitor  the  status  of  each  risk  periodically  and  implement  the  risk 
mitigation  plan  as  appropriate. 

Specifically,  we  analyzed  program  risk  documentation,  including  monthly 
risk  logs  and  reports,  risk-level  assignments,  risk  management  plans,  risk 
mitigation  plans,  and  risk  board  meeting  minutes.  Additionally,  we 
interviewed  program  officials  to  obtain  additional  information  about  their 
risks  and  risk  management  practices. 

To  address  the  fourth  objective,  we  used  DOD’s  official  list  of  MAIS 
programs,  as  of  February  25,  2015.  These  programs  were  used  as  basis 
to  determine  whether  programs  were  reported  to  the  Federal  IT 
Dashboard.  To  do  so,  we: 

•  Exported  the  IT  portfolio  program  data  reported  to  the  Federal  IT 
Dashboard  by  DOD  in  fiscal  year  2015,  and  compared  it  to  the  39 
programs  on  DOD’s  official  list  of  MAIS  programs. 

•  For  those  programs  that  were  found  to  not  be  reported  on  the 
Dashboard,  we  met  with  agency  officials  from  the  Office  of  the  Chief 
Information  Officer  (OCIO)  and  the  Under  Secretary  of  Defense 
(Acquisition,  Technology,  and  Logistics)  (AT&L)  to  determine  the 
reasons  for  not  reporting. 

We  also  interviewed  OCIO  officials  to  obtain  information  on  the  processes 
used  by  the  OCIO  when  reviewing  programs  for  IT  Dashboard  updates. 
Further,  we  interviewed  officials  at  OMB  to  obtain  their  views  on 
representing  the  CIO  as  the  sole  party  responsible  for  programs  reported 
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to  the  IT  Dashboard  was  accurate.  Specifically,  we  discussed  DOD’s 
unique  structure  that  AT&L  is  responsible  for  the  acquisition  performance 
of  MAIS  programs  but  is  a  separate  organization  from  the  OCIO. 

We  conducted  this  performance  audit  from  April  2015  to  March  2016  in 
accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 
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This  section  contains  profiles  of  the  three  selected  major  automated 
information  system  (MAIS)  programs  for  which  we  determined  whether 
they  had  changed  their  planned  cost  and  schedule  estimates  and  met 
performance  measures.  Each  profile  presents  data  on  the  program’s 
purpose  and  status,  its  latest  cost  and  schedule  estimates  compared  to 
the  first  acquisition  program  baseline  (where  established),  as  well  as 
system  performance  data.1 

The  first  page  of  each  two-page  profile  contains  a  description  of  the 
program’s  purpose  and  a  figure  that  provides  a  comparison  of  the 
program’s  first  acquisition  program  baseline  to  the  program’s  latest 
schedule.  The  years  depicted  on  the  figure  represent  calendar  years,  and 
the  milestones  represent  the  program’s  best  estimates  of  dates  for  those 
milestones.  The  program’s  start  represents  the  date  that  program  officials 
reported  that  they  first  started  work  on  the  program.2 

The  first  page  also  provides  (1)  essential  program  details,  such  as  the 
name  of  the  prime  contractor,  the  total  number  of  active  contractors — 
which  includes  the  prime  contractor — and  any  other  contractors  (and  in 
some  cases  subcontractors)  supporting  the  program;  (2)  program  costs 
(in  then-year  dollars),  comparing  the  program’s  latest  life-cycle  cost 
estimate  (separated  into  acquisition  and  operations  and  maintenance 
costs)  to  its  first  acquisition  program  baseline  (subsequent  acquisition 
program  baselines  that  may  have  been  established  are  not  identified),3 
(3)  locations  to  which  the  system  will  be  deployed;  and  (4)  a  summary  of 
the  cost,  schedule,  and  performance  of  each  program,  which  is  further 
discussed  on  the  second  page  of  the  profile.  The  symbols,  denoted  by 
arrows  or  filled  circles,  included  in  the  summary  box  on  the  first  page  of 
each  profile  and  in  the  headings  on  the  second  page  represent  whether  a 
program’s  cost  estimate  had  increased  (A),  decreased  (V),  or  stayed 
within  (i.e.,  not  to  exceed  threshold)  planned  cost  estimate  (i.e.,  (•)  and 


]A  program’s  first  acquisition  program  baseline  contains  the  original  life-cycle  cost 
estimate,  schedule  estimate,  and  performance  parameters  that  were  approved  for  that 
program  by  the  milestone  decision  authority.  The  first  acquisition  program  baseline  is 
established  after  the  program  has  assessed  the  viability  of  various  technologies  and 
refined  user  requirements  to  identify  the  most  appropriate  technology  solution  that 
demonstrates  that  it  can  meet  users’  needs. 

2This  date  is  different  than  what  DOD  considers  formal  program  initiation — the  date  that  a 
program  achieves  milestone  B. 

3An  estimate  in  then-year  dollars  includes  the  effects  of  economic  inflation. 
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whether  the  program’s  schedule  estimate  had  slipped  (>),been 
accelerated  to  meet  milestones  earlier  than  planned  (<),  or  stayed  within 
(i.e. ,  not  to  exceed  threshold)  planned  schedule  estimate  (•)  of  meeting 
milestones. 

The  second  page  of  each  profile  provides  detailed  information  on  each 
program’s  status,  costs,  schedule,  and  performance. 

•  In  the  status  section,  we  discuss  recent  and  upcoming  milestones  and 
events  for  each  program. 

•  In  the  cost  section,  we  identify  the  extent  to  which  the  program’s  life- 
cycle  cost  estimate  has  changed  from  its  first  acquisition  program 
baseline,  as  well  as  the  causes  for  any  changes  identified. 

•  In  the  schedule  section,  we  discuss  the  extent  to  which  the  program’s 
schedule  has  changed  from  its  first  acquisition  program  baseline,  and 
the  causes  for  any  schedule  changes  identified. 

•  In  the  performance  section,  we  identify  the  extent  to  which  each 
program  has  met  its  established  measures,  as  well  as  discuss  the 
results  of  system  performance  tests.  These  performance  ratings 
represent  a  point-in-time  assessment  as  reported  by  the  program. 
System  performance  targets  were  rated  as  “met”  when  (1)  system 
tests  were  passed  with  no  deficiencies  or  limitations,  (2)  the  program 
fully  met  all  of  its  key  performance  parameters,  or  (3)  a  program  had 
addressed  all  deficiencies  or  limitations  that  were  identified  during 
system  tests.  System  performance  was  rated  as  “not  fully  met”  when 
a  program  either  (1)  did  not  fully  pass  system  testing  and  was  still  in 
the  process  of  addressing  the  deficiencies  or  limitations  identified 
during  system  testing  or  (2)  did  not  pass  system  testing  and 
subsequently  removed  the  problematic  functionality  from  the  system 
in  order  to  pass  subsequent  system  tests,  instead  of  fixing  the 
problematic  functionality  and  keeping  it  in  the  planned  release  of  the 
system. 

Army  Tactical  Mission  Command  (TMC) 

TMC  is  a  suite  of  products — comprised  of  hardware  and  software 
equipment  and  elements — that  are  intended  to  provide  the  Army 
commanders  and  their  staff  with  mission  command  capabilities,  such  as 
real-time  situational  awareness  and  a  user-defined  common  operational 
picture.  TMC  products  are  fielded  worldwide  and  are  intended  to  support 
decision-making,  planning,  rehearsal,  and  execution  management.  TMC 
is  now  engaged  in  transitioning  to  the  web-based  Command  Post 
Computing  Environment.  One  key  element  of  this  new  environment  — 
known  as  Tactical  Applications — is  aimed  to  minimize  administrative 
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burdens  on  the  user,  and  simplify  the  overall  Mission  Command 
collaborative  experience. 


First  acquisition  program  baseline  as  of  February  2008 
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Program  Essentials  (as  of  October  2015) 

DOD  component:  Department  of  the  Army 

Program  owner:  Project  Manager  Mission  Command, 
Program  Executive  Office  Command,  Control,  and 
Communications-Tactical 

Prime  contractor:  Weapons  System  Engineering  Center 
Total  number  of  contractors:  not  applicable 
Fiscal  year  2015  funding  requested:  $146.5  million 


Deployment  Details  (as  of  October  2015) 

Current  number  of  expected  users:  not  applicable 

Current  number  of  expected  locations:  varies  tactical  units 
based  on  Army  fielding  prioritization  directives 

Legacy  system(s)  to  be  replaced:  0 

Annual  cost  of  legacy  system(s):  $0 

Number  of  system  interfaces:  0 


Program  Costs  (then-year  dollars  in  millions) 


First 

Latest 

APB 

Estimate 

(02/2008) 

(01/2016) 

Life-cycle  cost  estimate 

$1,969.0 

$2,337.1 

-  Acquisition 

1,852.5 

2,224.2 

-  Operations  and  maintenance 

116.5 

112.9 

Amount  spent  to-date  $1,465.64 

(as  of  October  2015) 

Cost,  Schedule,  and  Performance  Summary 

»  Exceeded  planned  cost  estimate  (A) 

»  Stayed  within  planned  schedule  estimate  (•) 

»  Met  system  performance  targets 
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Program  Status 

All  of  the  products  included  in  TMC  are  post-development  and  in 
production.  The  program  continues  to  field  equipment  and  perform  a 
technical  refresh  of  the  hardware  and  software  in  the  field.  The  program  is 
also  working  to  resolve  sustainment  metric  issues  and  updating  their  life- 
cycle  sustainment  plan. 

Exceeded  Planned  Cost  Estimate  (A) 

TMC’s  planned  total  life-cycle  cost  estimate  has  increased  by  19  percent 
from  the  program’s  first  acquisition  program  baseline  estimate  of 
approximately  $1.97  billion.  Specifically,  as  of  January  2016,  the  life-cycle 
cost  estimate  was  approximately  $2.34  billion.  Program  officials  reported 
that  the  increased  costs  were  attributed  to  research  and  development 
testing  and  evaluation  cost  estimate  breach  that  was  reported  to 
Congress.  The  Army’s  TMC  program  estimated  program  development 
cost  increased  by  45  percent  over  the  original  estimate  due  to  program 
scope  changes  derived  from  the  realignment  of  Command  Post  of  the 
Future  as  a  foundation  for  Mission  Command  Collapse,  the  integration  of 
Personalized  Assistant  that  Learns,  and  the  incorporation  of  future  force 
requirements. 

Stayed  within  Planned  Schedule  Estimate  (•) 

As  of  January  2016,  the  program  had  experienced  a  3  month  slippage  in 
its  full  deployment  date  compared  to  its  first  acquisition  program  baseline 
of  September  2018.  The  slippage  was  within  the  pre-established 
threshold  allowance  to  account  for  minor  shifts  in  program  schedule. 
Program  officials  stated  that,  although  the  Command  Post  of  the  Future 
product  is  95  percent  fielded  and  is  on  schedule  to  reach  full  deployment 
by  December  2018,  continued  support  of  the  Command  Post  Computing 
Environment  is  needed  beyond  fiscal  year  2019.  Program  officials 
considered  the  slippage  to  be  of  low  risk  and  will  continue  to  operate 
within  the  planned  schedule  estimate. 

Met  System  Performance  Targets 

As  of  January  2016,  the  TMC  program  met  all  three  of  its  key 
performance  parameters,  including  net-centric  military  operations, 
disseminate  orders  with  future  Army  and  Joint  C2  systems,  and 
displaying  unified  information  on  subject  matters,  such  as  friendly  and 
enemy  forces. 
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Navy  Common  Aviation  Command  and  Control  System  (CAC2S) 
Increment  1 

CAC2S  is  an  integrated  and  coordinated  modernization  effort  for  the 
equipment  of  the  Marine  Air  Command  and  Control  System  and  is 
intended  to  provide  enhanced  capability  for  three  defense  centers  to 
support  aviation  employment  in  joint,  combined,  and  coalition  operations. 
CAC2S  provides  the  tactical  situational  display,  information  management, 
sensor  and  data  link  interface,  and  operational  facilities  for  planning  and 
execution  of  Marine  Aviation  missions  within  the  Marine  Air  Ground  Task 
Force.  It  is  intended  to  replace  existing  aviation  command  and  control 
equipment  from  12  legacy  systems.  CAC2S  Increment  1  will  eliminate  the 
Air  Command  and  Control  systems  and  will  capability  for  aviation  combat 
direction  and  air  defense  functions  by  providing  a  single  networked 
system. 
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First  acquisition  program  baseline  schedule  as  of  August  2000 


Technology  development 
Planned 

Source:  GAO  analysis  of  agency  data.  |  GAO-16-336 


Materiel  solution  analysis 
Actual 
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manufacturing  development 
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Program  Essentials  (as  of  October  2015) 

DOD  component:  Department  of  the  Navy 

Program  owner:  Program  Executive  Officer  Land  Systems 
(United  States  Marine  Corps) 

Prime  contractor:  General  Dynamics 

Total  number  of  contractors:  5 

Fiscal  year  2015  funding  requested:  $53.9  million 


Deployment  Details  (as  of  October  2015) 

Current  number  of  expected  users:  22  of  1 68 
Current  number  of  expected  locations:  11  of  11 

Legacy  systems  to  be  replaced:  12 
Annual  cost  of  legacy  systems:  $36.7  million 
Number  of  system  interfaces:  12 


Program  Costs  (then-year  dollars  in  millions) 

Latest 


First  APB 

Estimate 

(08/2000) 

(10/2015) 

Life-cycle  cost  estimate 

$347.0 

$2002.6 

-  Acquisition 

173.4 

912.0 

-  Operations  and  maintenance  173.6 

1,090.6 

Amount  spent  to-date  $609.1 

(as  of  October  2015) 

Cost,  Schedule,  and  Performance  Summary 

»  Exceeded  planned  cost  estimate  (A) 

»  Exceeded  planned  schedule  estimate  (>) 

»  Met  system  performance  targets 
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Program  Status 

CAC2S  Increment  1,  which  comprises  of  two  phases,  is  currently  in  post¬ 
milestone  C  where  the  phase  2  system  is  now  in  development.  CAC2S’s 
current  work  consists  of  developmental  testing  and  the  production  of  the 
limited  deployment  units  in  preparation  for  the  March  2016  Initial 
Operational  Test  and  Evaluation.  To  help  achieve  its  goals  of  a 
successful  Initial  Operation  Test  and  Evaluation,  a  phase  2  milestone  C 
decision  was  authorized  in  February  2015  for  CAC2S  to  procure  four 
limited  deployment  units.  As  of  October  2015,  CAC2S  had  delivered  all 
four  limited  units  to  support  current  developmental  testing.  Production  of 
the  limited  deployments  units  are  on  schedule  and  planning  activities  are 
underway.  Regarding  the  developmental  testing,  program  officials 
anticipate  favorable  test  results  after  subsequent  software  enhancements 
had  been  made  to  address  software  concerns  identified  in  prior 
developmental  testing. 

Exceeded  Planned  Cost  Estimate  (A) 

As  of  October  2015,  CAC2S’s  life  cycle  cost  estimate  was  $2  billion, 
which  was  about  a  477  percent  increase  from  its  first  acquisition  program 
baseline  estimate  of  $347  million  established  in  August  2000.  As 
previously  reported,  factors  that  attributed  to  the  cost  increase  were  early 
challenges  in  estimating  costs  due  to  program  scope  growth  and 
restructuring.  According  to  program  documentation,  operations  and 
support  expenditures  of  approximately  $1.6  billion  for  the  production  of 
milestone  C  had  been  carried  over  into  the  program’s  total  life-cycle  cost 
estimate.  However,  since  our  previous  report,  program  documentation 
indicated  that  improvements  to  the  cost  position  are  being  made. 
Specifically,  the  milestone  C  service  cost  position,  dated  February  2015, 
produced  a  cost  avoidance  of  $54.4  million  compared  to  its  2010  cost 
assessment.  As  of  October  2015,  the  program’s  latest  life-cycle  estimate 
relative  to  its  November  2010  production  acquisition  program  baseline 
cost  estimate  had  decreased  about  19  percent.  Program  officials 
attributed  this  decrease  due  to  the  program  embracing  the  DOD  Better 
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Buying  Power  initiatives4  that  benefitted  from  competitive  market  forces 
that  drove  down  cost. 

Exceeded  Planned  Schedule  Estimate  (>) 

As  of  October  2015,  CAC2S  Increment  1’s  estimated  full  deployment  date 
was  March  2022,  which  represented  a  13  year  and  9  month  schedule  slip 
from  the  program’s  first  acquisition  program  baseline  schedule  estimate. 
As  previously  reported5,  factors  that  attributed  to  the  schedule  delay 
included  the  addition  of  new  requirements  and  program  restructure. 
Program  officials  stated  that  subsequent  to  our  prior  report,  the  program 
has  been  executing  in  accordance  within  its  approved  schedule.  As  of 
October  2015,  the  estimated  milestone  C  phase  2  was  delayed  by  6 
months  from  the  program’s  production  acquisition  program  baseline 
schedule  but,  as  stated  above,  achieved  its  milestone.  Program  officials 
attributed  this  delay,  in  part,  to  administrative  factors,  which  included  the 
review  and  approval  process  of  getting  signature  approval.  CAC2S 
successfully  achieved  milestone  C  phase  2  approval  in  February  2015  but 
the  acquisition  decision  memorandum  had  not  been  signed  until  March 
2015. 

Met  System  Performance  Targets 

As  of  October  2015,  CAC2S  program  documentation  reported  that  it  was 
meeting  both  of  its  key  performance  parameters  related  to  net-ready  and 
data  fusion.  Program  officials  stated  that,  during  testing  of  the  program’s 
key  performance  parameters,  its  net-ready  and  data  fusion  performance 
targets  were  both  met,  while  many  attributes  for  the  data  fusion  key 
performance  parameter  were  consistently  above  the  threshold  for  being 
met. 


4DOD  Better  Buying  Power  initiative  is  the  implementation  of  best  practices  to  strengthen 
the  department’s  buying  power,  improve  industry  productivity,  and  provide  an  affordable, 
value-added  military  capability  to  the  warfighter. 

5GAO-1 3-311. 
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Air  Force  Defense  Enterprise  Accounting  and  Management  System 
(DEAMS)  Increment  1 

The  DEAMS  Increment  1  program  is  intended  to  provide  the  Air  Force 
with  the  entire  spectrum  of  financial  management  capabilities,  including 
collections;  commitments  and  obligations;  cost  accounting;  general 
ledger;  funds  control;  receipts  and  acceptance;  accounts  payable  and 
disbursement;  billing;  and  financial  reporting  DEAMS  is  also  intended  to 
be  a  key  component  of  DOD’s  solution  for  achieving  fully-auditable 
financial  statements  by  September  30,  2017,  as  required  by  the  National 
Defense  Authorization  Act  for  Fiscal  Year  2010. 
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First  acquisition  oroaram  baseline  as  of  February  2012 


1  Full  deployment1 


ITTIT1TTTTTTT1 

1  1  1  1  1 1  1  1  1 1 II 1 1 1 1 1  1  1 1  1  1 1 

111111111111 

1111111111111 

1111111111111 

1111111111111 

111111111111 

1111111111111 

1  1  1  1  1  1  1  1  1  1  1 

111111111111 

III  1  1  1  1  1  1  1  1  1  II 

1111111111 

111111111111111 

I  2003  I 

2004  |  2005  | 

2006 

2007 

2008 

2009 

2010 

2011  | 

2012 

]  2013 

2014 

2015 

2016  | 

Materiel  solution  analysis 

Technology  development  Engineering  development 

o 

Actual 

@ 

Planned 

Full  deployment 


Source:  GAO  analysis  of  agency  data.  |  GAO-16-336 


Program  Essentials  (as  of  October  2015) 

DOD  component:  Department  of  the  Air  Force 

Program  owner:  Assistant  Secretary  for  Financial 
Management  and  Comptroller 

Prime  contractor:  Accenture 

Total  number  of  contractors:  29 

Fiscal  year  2015  funding  requested:  $171 .99  million 


Program  Costs  (then-year  dollars 

in  millions) 

Latest 

First  APB 

Estimate 

(02/2012) 

(10/2015) 

Life-cycle  cost  estimate 

$1,434.3 

$1563.6 

-  Acquisition 

853.2 

868.7 

-  Operations  and  maintenance  581.1 

694.9 

Amount  spent  to-date 

$670.6  million 

(as  of  October  2015) 

Deployment  Details  (as  of  October  2015)  Cost,  Schedule,  and  Performance  Summary 

Current  number  of  expected  users:  14,300  of  23,500  »  Exceeded  planned  cost  estimate  (A) 

Current  number  of  expected  locations:  102  of  164  »  Exceeded  planned  schedule  estimate  (>) 

Legacy  systems  to  be  replaced:  2  fully  and  1 0  partially  »  Did  not  fully  meet  system  performance  targets 

Annual  cost  of  legacy  systems:  approximately  $50  million 
Number  of  expected  system  interfaces:  95 
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Program  Status 

As  of  November  2015,  the  DEAMS  program  is  working  to  achieve  full 
deployment  decision  by  February  2016.  In  August  2015,  the  initial 
operational  test  and  evaluation  report,  conducted  by  the  Air  Force 
Operational  Test  and  Evaluation  Center,  indicated  a  number  of  findings 
requiring  remediation  prior  to  the  February  2016  full  deployment  decision. 
Nevertheless,  DEAMS  was  granted  a  limited  deployment  decision,  but  the 
program  experienced  a  significant  change  as  a  result  of  breaching  the  full 
deployment  decision  threshold  for  a  timing  issue  only.  DEAMS  current 
work  efforts  consist  of  deployment  to  new  users  and  remaining  35  sites, 
capability  development  for  deployment,  training  new  users,  and  resolving 
initial  operational  test  and  evaluation  findings. 

Exceeded  Planned  Cost  Estimate  (A) 

As  of  October  2015,  DEAMS’s  latest  life-cycle  cost  estimate  was  about 
$1 .56  billion,  which  was  about  a  9  percent  increase  from  its  first 
acquisition  program  baseline  estimate  of  approximately  $1 .43  billion — 
established  in  February  2012.  Program  officials  attributed  this  increase,  in 
part,  to  program  scope  growth  due  to  addition  of  requirements  from 
increment  2  and  the  addition  of  a  second  Oracle  software  upgrade 
projected  for  2021 . 

Exceeded  Planned  Schedule  Estimate  (>) 

DEAMS  experienced  a  6  month  slippage  in  its  milestone  C  but 
successfully  attained  milestone  C  approval  within  the  established 
threshold.  Program  officials  did  not  provide  a  rationale  for  factors  that 
attributed  to  this  delay  but  maintained  that  the  program  operated  within 
the  threshold  requirements.  DEAMS  also  experienced  a  1  year  slippage 
in  its  full  deployment  decision  date — currently  scheduled  for  February 
2016.  Program  officials  attributed  this  slippage  due  to  findings  identified 
DEAMS’s  initial  operational  test  and  evaluation  report. 

Did  Not  Fully  Meet  System  Performance  Targets 

As  of  October  2015,  DEAMS  program  officials  reported  that  it  did  not 
meet  all  of  its  nine  key  performance  parameters.  Specifically,  DEAMS  did 
not  meet  five  key  performance  parameters:  Balance  with  Treasury, 
Accurate  Balance  of  Available  Funds,  Timely  Report,  Period-End 
Processing,  and  Net-Ready.  For  example,  an  initial  operational  test  and 
evaluation  report,  listed  above,  identified  system  performance  issues, 
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which  included  unstable  change  management  issues,  transaction 
backlogs,  and  ineffective  reporting  tools.  Subsequently,  the  Air  Force 
Operational  Test  and  Evaluation  Center  provided  29  recommendations 
for  the  Air  Force  to  implement  to  support  the  successful  fielding  of 
DEAMS  Increment  1,17  of  which  were  documented  as  being  completed, 
while  corrective  action  for  the  remaining  12  are  still  underway.  The 
program  is  expected  to  demonstrate  improvement  before  it  will  be 
authorized  to  be  deployed  to  all  users. 


Page  48 


GAO-16-336  Major  Automated  Information  Systems 


Appendix  III:  Comments  from  the 
Department  of  Defense 


ASSISTANT  SECRETARY  OF  DEFENSE 
3600  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3600 

MAR  15  2016 


Ms.  Carol  R.  Cha 
Director,  Information  Technology 
Acquisition  Management  Issues 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 

Dear  Ms.  Cha: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  Government  Accountability 
Office  (GAO)  Draft  Report,  GAO-16-336.  “DOD  MAJOR  AUTOMATED  INFORMATION 
SYSTEMS:  Improvements  Can  Be  Made  in  Reporting  Critical  Changes  and  Clarifying 
Leadership  Responsibility"  dated  February  16.  2016  (GAO  Code  100097).  Detailed  comments 
on  the  report  recommendations  are  enclosed. 


Sincerely, 

Mr.  James  A.  MacStravic 
Acting  Principal  Deputy  Assistant  Secretary  of 
Defense  for  Acquisition 

Enclosure: 

As  stated 


ACQUISITION 
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GAO  Draft  Report  Dated  February  16,  2016 
GAO-16-336  (GAO  CODE  100097) 

“DOD  MAJOR  AUTOMATED  INFORMATION  SYSTEMS:  IMPROVEMENTS  CAN 
BE  MADE  IN  REPORTING  CRITICAL  CHANGES  AND  CLARIFYING  LEADERSHIP 

RESPONSIBILITY” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATION 


RECOMMENDATION  1 :  To  help  improve  the  management  of  MA1S  programs,  GAO 
recommends  that  the  Secretary  of  Defense,  along  with  relevant  departments,  examine  the  MAIS 
critical  change  reporting  process  to  identify  root  causes  for  delays  and  implement  corrective 
actions  for  the  timely  delivery  of  critical  change  reports. 

DoD  RESPONSE:  Concur.  The  Department  will  conduct  a  root  cause  analysis  based  on  a 
sampling  of  the  CCRs  and  apply  lessons  learned  to  future  CCRs. 

RECOMMENDATION  2:  To  help  improve  the  management  of  MAIS  programs,  GAO 
recommends  that  the  Secretary  of  Defense,  along  with  relevant  departments,  develop  a 
mechanism  for  monitoring  whether  MAIS  programs  with  late  reports  are  restricted  from 
obligating  funds  and  in  turn  ensuring  compliance  with  the  Antideficiency  Act. 

DoD  RESPONSE:  Concur.  The  Department  will  recommend  Senior  Officials  establish  controls 
to  monitor  program  obligations  for  MAIS  programs  with  late  reports. 

RECOMMENDATION  3:  To  help  improve  the  management  of  MAIS  programs,  GAO 
recommends  that  the  Secretary  of  the  Army  direct  the  TMC  program  manager  to  develop  a 
requirements  management  plan  to  document  and  manage  its  requirements  process. 

DoD  RESPONSE:  Concur.  The  program  has  an  established  requirements  database  that  it 
currently  uses.  They  will  expand  it  to  meet  the  intent  of  a  formal  Requirements  Management 
Plan.  In  this  way  it  will  provide  a  mechanism  to  more  accurately  track  and  ensure  that  project 
plans,  activities  and  work  products  are  consistent  with  defined  requirements. 

RECOMMENDATION  4:  To  help  improve  the  management  of  MAIS  programs.  GAO 
recommends  that  the  Secretary  of  the  Navy  direct  the  CAC2S  program  manager  to  identify 
weaknesses  in  the  requirements  traceability  process  and  take  corrective  actions  to  manage  the 
traceability  of  requirements  to  the  respective  lower-level  requirements,  and  periodically  evaluate 
work  products,  including  the  requirements  management  plan,  and  update  them  in  accordance 
with  the  requirements  guidance. 

DoD  RESPONSE:  Concur.  The  Navy  has  corrected  the  data  query  issue  that  caused  1 1 
requirements  to  be  omitted  from  the  traceability  matrix  provided  to  the  Government 
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Accountability  Office.  The  Navy  has  also  identified  the  weakness  in  the  traceability  process  that 
led  to  14  general  CAC2S  requirements,  e.g.,  lighting  and  acoustic  noise,  not  being  fully  traced. 

RECOMMENDATION  5:  To  help  improve  the  management  of  MAIS  programs,  GAO 
recommends  that  the  Secretary  of  the  Air  Force  direct  the  DEAMS  program  manager  to  address 
weaknesses  in  its  controls  for  ensuring  that  all  software  requirements  are  tested  and  validated 
before  deployment  of  new  software  releases. 

DoD  RESPONSE:  Concur.  DoD  Instruction  5000.02,  Operation  of  the  Defense  Acquisition 
System,  requires  testing  and  validation  of  requirements  before  deployment  of  software  releases. 
The  Milestone  Decision  Authority  is  responsible  for  considering  program  risk  prior  to  approving 
deployments. 
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GAO  Contact 


Carol  R.  Cha  at  (202)  512-4456  or  ChaC@gao.gov 


Staff 

Acknowledgments 


In  addition  to  the  contact  name  above,  the  following  staff  also  made  key 
contributions  to  this  report:  Eric  Winter,  Assistant  Director;  Ronalynn 
(Lynn)  Espedido;  Corey  Evans;  Rebecca  Eyler;  Franklin  Jackson;  Kate 
Nielsen;  John  Ortiz;  Kathleen  Sharkey;  and  Jeanne  Sung. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and 
policies;  and  provides  analyses,  recommendations,  and  other  assistance 
to  help  Congress  make  informed  oversight,  policy,  and  funding  decisions. 
GAO’s  commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no 
cost  is  through  GAO’s  website  (http://www.gao.gov).  Each  weekday 
afternoon,  GAO  posts  on  its  website  newly  released  reports,  testimony, 
and  correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted 
products,  go  to  http://www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  website, 
http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 
MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

Connect  with  GAO 

Connect  with  GAO  on  Facebook,  Flickr,  Twitter,  and  YouTube. 

Subscribe  to  our  RSS  Feeds  or  E-mail  Updates. 

Listen  to  our  Podcasts  and  read  The  Watchblog. 

Visit  GAO  on  the  web  at  www.gao.gov. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Website:  http://www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Katherine  Siggerud,  Managing  Director,  siggerudk@gao.gov,  (202)  512- 
4400,  U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room 

71 25,  Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  youngd@gao.gov,  (202)512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 
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